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PREFACE 


The  research  documented  in  this  technical  report  was  sponsored  by  the  Air  Force 
Research  Laboratory,  Deployment  and  Sustainment  Division,  Logistics  Readiness 
Branch.  This  volume  is  the  third  of  three  volumes  that  summarize  work  performed  to 
develop  an  Aircraft  Battle  Damage  Assessment  and  Repair  (ABDAR)  technology  to 
enhance  the  capability  of  Air  Force  technicians  to  assess  damage,  determine  needed 
repairs  and  restore  the  aircraft  to  operational  status.  The  work  was  funded  under 
PE63106F,  Project  2745.  The  work  was  performed  under  contract  F41624-95-C-5003 
by  NCI  Information  Systems,  Inc.,  with  subcontractor  support  from  Boeing  Aircraft 
Company,  RJO  Enterprises,  Inc.,  and  GRACAR  Corporation.  Captain  Michael  Clark 
and  1st  Lieutenant  Steve  Grace  were  the  program  managers  for  the  major  portion  of  the 
effort.  Other  Laboratory  personnel  who  made  major  contributions  earlier  in  the  program 
were  Captain  Eric  Carlson,  Captain  Floyd  Gwartney,  1st  Lieutenant  J.C.  Bradford,  and 
1st  Lieutenant  Maurice  Azar. 

This  research  could  not  have  been  accomplished  without  the  support  and  assistance 
of  many  members  of  the  Combat  Logistics  Support  Squadrons,  the  Aircraft  Battle 
Damage  Repair  Program  Office,  and  the  Air  Force  Materiel  Command  Logistics 
Directorate  who  served  as  members  of  the  ABDAR  Users  Group,  provided  technical 
guidance  throughout  the  program,  and  provided  program  advocacy. 

The  653^''  Combat  Logistics  Support  Squadron,  Robins  AFB  provided  extraordinary 
support  for  the  program.  The  653’'^'  provided  the  test  facilities,  test  aircraft,  and  many  of 
the  technicians  who  participated  in  the  field  test.  The  squadron  also  provided  the 
support  of  several  of  their  instructors  who  served  as  subject  matter  experts  and  advisors 
throughout  the  program.  The  contributions  of  MSgt  Ken  McCain,  TSgt  Geoffrey  Miller, 
TSgt  George  Boutwell,  TSgt  Ken  Dockery,  and  TSgt  Rob  Meyers  as  technical  advisors 
were  invaluable  and  greatly  appreciated  by  the  ABDAR  program  staff. 

The  Field  Test  Report  is  the  third  volume  of  a  three-volume  final  program  report.  It 
describes  the  objectives,  methodology,  and  results  of  the  ABDAR  field  test. 
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SUMMARY 


The  objective  of  the  Aircraft  Battle  Damage  Assessment  and  Repair  (ABDAR) 
program  was  to  develop  and  demonstrate  technology  to  provide  Aircraft  Battle  Damage 
Repair  (ABDR)  technicians  and  assessors  with  ready  access  to  specialized  tools  and 
the  technical  information  required  to  perform  their  jobs.  The  ABDAR  program 
developed  and  field-tested  the  techniques  and  technologies  required  to  implement  an 
operational  ABDAR  System.  Technological  developments  include  advances  in  the 
areas  of  Interactive  Electronic  Technical  Manuals  (lETMs),  computer-generated 
diagnostics,  human/computer  interface,  portable  maintenance  aids  (PMAs),  and  the 
use  of  complex  databases.  These  advances  were  incorporated  into  the  ABDAR 
Demonstration  System  for  evaluation  under  simulated  field  conditions. 

This  volume  of  the  technical  report  describes  the  results  of  a  field  test  conducted  to 
evaluate  the  ABDAR  Demonstration  System.  The  system  was  evaluated  by  comparing 
the  performances  of  technicians  performing  a  simulated  aircraft  battle  damage 
assessment  task  using  the  ABDAR  demonstration  system  with  the  performances  of 
technicians  performing  the  task  with  conventional  paper  based  technical  orders  (TOs). 
The  study  compared  two  types  of  electronic  technical  data  presented  on  the  ABDAR 
Demonstration  System  (Content  Data  Model  [CDM],  Indexed  Portable  Document 
Format  [IPDF])  with  the  performance  of  technicians  using  paper  TOs.  Dependent 
variables  of  speed,  accuracy,  and  completeness  were  measured.  Subjects  using  the 
ABDAR  Demonstration  System  with  CDM  data  performed  significantly  faster  than 
subjects  using  the  ABDAR  Demonstration  System  with  IPDF  data,  improving  the  overall 
time  by  86%.  Subjects  using  CDM  and  IPDF  data  were  significantly  more  accurate  and 
complete  than  subjects  using  Paper,  regardless  of  technician  type.  Assessments  by 
subjects  using  CDM  were  39%  more  complete  and  51%  more  accurate  than 
assessments  conducted  by  subjects  using  paper  TOs.  Assessments  by  subjects  using 
IPDF  were  34%  more  complete  and  44%  more  accurate  than  subjects  using  Paper 
TOs.  Overall,  the  ABDAR  Demonstration  System  tools,  in  conjunction  with  electronic 
technical  data,  provided  a  significant  advantage  over  the  current,  paper-based  method 
of  performing  ABDR  assessments. 

In  addition  to  the  demonstrated  performance  enhancements  to  ABDR,  the  ABDAR 
Demonstration  System  has  a  high  rate  of  acceptance  among  the  potential  users.  Users 
expressed  a  strong  desire  to  have  an  ABDAR  system  implemented  for  operational  use. 
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AIRCRAFT  BATTLE  DAMAGE  ASSESSMENT  AND  REPAIR 

(ABDAR) 

FINAL  PROGRAM  REPORT 
VOLUME  3:  FIELD  TEST  REPORT 

INTRODUCTION 

The  objective  of  the  ABDAR  program  was  to  develop  an  ABDAR  Demonstration 
System  that  would  significantly  enhance  the  speed,  accuracy,  and  completeness  of  the 
assessment  of  damage  inflicted  on  an  aircraft  during  combat  operations.  The  ABDAR 
Demonstration  System  developed  in  the  ABDAR  program  was  an  end-to-end  system. 
The  process  started  at  aircraft  Debrief  and  finished  with  final  documentation  of  the 
damage  on  an  Air  Force  Technical  Order  (AFTO)  Form  97.  The  ABDR  process  and 
requirements  were  supported  with  technical  data  from  applicable  ABDR  manuals  for  a 
range  of  aircraft  systems,  including  TO  1-1H-39,  Technical  Manual,  General  Aircraft 
Battle  Damage  Repair.  The  system  supports  two  different  Electronic  Technical 
Information  (ETI)  types.  Integrated  Portable  Document  File  (I PDF)  and  Content  Data 
Model  (CDM).  This  volume  reports  the  results  of  a  field  test  to  evaluate  the  ABDAR 
Demonstration  System. 

Data  collection  was  conducted  by  AFRL/HESR,  with  technical  and  administrative 
support  provided  by  NCI  and  Boeing.  Analysis  of  data  and  report  writing  for  the  test 
was  accomplished  by  NCI.  The  field  test  provided  a  structured  approach  to  the  testing 
of  the  ABDAR  Demonstration  System.  The  field  test  was  conducted  at  Robins  AFB 
between  September  1998  and  October  1999.  The  test  consisted  of  two  distinct  phases: 
Phase  I  -  ABDAR  process  Paper  Technical  Data  version  and  Phase  II  -  ABDAR 
process  ETI  version.  This  report  presents  the  findings  from  the  field  test  and  provides 
recommendations  for  implementation  of  the  ABDAR  technology. 

Background 

The  ABDAR  program  was  an  advanced  development  research  and  development 
(R&D)  project  under  the  sponsorship  of  Air  Force  Research  Laboratory's  Logistics 
Readiness  Branch  (AFRL/HESR).  The  requirement  was  to  develop  and  demonstrate 
technology  that  would  provide  a  significant  enhancement  in  the  capability  of  USAF 
ABDR  assessors  and  technicians  to  rapidly  assess  battle  damaged  aircraft.  These 
individuals  face  the  critical  task  of  assessing,  repairing,  and  returning  battle-damaged 
aircraft  to  mission  readiness  during  wartime.  AFRL/HESR  and  the  USAF  and 
Department  of  Defense  (DoD)  ABDR  communities  have  long  recognized  that 
enhancements  to  this  capability  are  critical  to  success  in  future  armed  conflicts.  The 
development  of  an  enhanced  aircraft  battle  damage  repair  (ABDR)  assessment 
capability  will  provide  an  effective  force  multiplier  to  the  Combat  Air  Forces  (CAF). 


1 


The  55-month  ABDAR  program  supported  a  concept  developed  in  the  early  1990’s 
by  AFRL/HESR.  A  preliminary  demonstration  of  the  concept  was  developed  in  1994. 
The  preliminary  demonstration  effort  focused  on  devising  a  process  to  enhance  the 
assessment  and  repair  methodology  within  the  Integrated  Maintenance  Information 
System  (IMIS)  program  (Ward,  et  al.  1995,  Volumes  1,  2,  3,  and  Thomas,  1995).  The 
preliminary  demonstration  was  intended  to  be  used  as  a  module  that  would  be  tailored 
specifically  for  the  assessor’s  use  in  an  IMIS  environment.  A  principle  of  IMIS  was  the 
integration  of  multiple  sources  of  maintenance  information.  This  concept  was  a  guiding 
principle  for  the  ABDAR  program.  The  development  challenge  was  to  provide  that 
information  through  a  common  user  interface  that  operates  off  a  workstation  or  a 
portable  maintenance  aid  (PMA)  and  effectively  provides  the  ABDR  assessor  with  all  of 
the  information  required  to  perform  the  damage  assessment. 

The  preliminary  ABDAR  concept  demonstration  was  presented  at  an  ABDR  “Live- 
Fire”  Demonstration  Exercise  conducted  at  Davis-Monthan  AFB,  Arizona  in  October- 
November  1994.  The  AF  and  DoD  ABDR  User  Community  representatives  who  viewed 
the  demonstration  were  highly  receptive  to  the  AFRL  concept  and  encouraged  further 
development  of  the  concept  and  technology. 

That  fundamental  ABDAR  research  evolved  into  this  technology  development  effort. 
The  approach  was  to  perform  a  requirements  analysis  that  would  feed  “as  is”  and  “to 
be”  modeling  data  and  system  requirements  into  the  design,  development,  data 
authoring,  integration,  and  testing  of  an  ABDAR  Demonstration  System.  Those 
processes  started  in  August  1995  and  culminated  with  the  development  of  the  ADBAR 
Demonstration  System  for  use  at  the  field  test  in  1999-2000.  Throughout  the  program, 
AF  and  DoD  users  from  the  ABDR  community  were  actively  involved  in  the 
development  of  the  ABDAR  Demonstration  System  and  the  planning  and 
implementation  of  the  field  test.  The  field  test  focused  on  the  “assessment”  portion  of 
the  ABDR  process.  The  demonstration  system  developed  provided  the  means  to 
support  multiple  levels  of  ETI  including  Level  II  (IPDF  type  data)  and  Level  IV,  (CDM 
type  data).  Development  of  the  ABDAR  Demonstration  System  is  described  in  detail  in 
Volume  2  of  this  report. 

Field  Test  Goals 

The  goal  of  the  ABDAR  field  test  was  to  evaluate  the  ABDAR  concept,  as 
implemented  in  the  ABDAR  Demonstration  System,  in  a  simulated  ABDR  environment. 
Specifically,  the  field  test  was  designed  to  answer  the  following  questions. 

a.  Can  ABDR  assessors  more  effectively  perform  the  damage  assessment  and 
repair  planning  processes  when  using  the  ABDAR  Demonstration  System  than 
when  using  paper  technical  orders? 

b.  Are  technicians  who  perform  the  ABDR  assessment  and  repair  planning 
processes  more  effective  when  using  ABDAR  Demonstration  System  with  CDM 
data  than  technicians  using  the  system  with  IPDF  data? 


c.  is  there  a  difference  in  the  performance  of  F-15  ABDR  qualified  assessors  and 
the  performance  of  ABDR  assessors  qualified  on  other  aircraft? 

d.  Can  F-15  mechanics  perform  the  assessment  process  on  the  F-15  as  efficiently 
as  F-15  and  other  aircraft  qualified  assessors? 


Programmatic  Parameters 

The  following  field  test  parameters  are  provided  as  an  overview  of  the  field  test. 

a.  The  field  test  was  conducted  by  AFRL/HESR  with  contractor  support  from  NCI. 

b.  The  field  test  spanned  a  period  of  one  year,  with  actual  data  collection 
consuming  nine  weeks  of  that  time.  Three  weeks  of  testing  occurred  for  each  of 
the  three  media  types. 

c.  An  F-15A  aircraft  was  available  for  the  duration  of  the  testing.  The  damages, 
previously  inflicted  on  the  aircraft,  remained  unchanged  for  all  subjects.  The 
damages  inflicted  on  the  aircraft  accurately  simulate  typical  damage  received  in 
combat  operations. 

d.  Dedicated  hangar  space  was  available  for  use  in  all  data  collection  activities. 
The  hangar  space  was  the  same  for  the  pre-field  test  and  the  field  test. 

e.  A  pre-field  test  was  conducted  before  each  phase  of  the  actual  test.  Its  purpose 
was; 

(1)  To  verify  the  test  procedures. 

(2)  To  confirm  that  all  system  components  functioned  as  planned. 

(3)  To  train  the  evaluation  team  members  on  the  use  of  the  system  and  data 
collection  procedures. 

Field  Test  Hardware  and  Software  Requirements 

The  field  test  implementation  of  the  ABDAR  Demonstration  System  required  three 
integrated  hardware  segments.  See  volume  2,  Program  Methodology,  for  specific 
details  on  the  system  architecture.  See  Figure  1  below  for  a  graphic  representation  of 
the  system. 
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ABDAR_PMA1 
Dell  Lattitude  Notebook 
Windows  NT  4.0 
Workstation 


Dell  Dimension  XPS  T450 
V\findows  98 


Figure  1  -  Field  Test  Hardware  Design 

a.  PMA  -  a  laptop  computer  used  to  provide  technical  information  for  use  by 
assessors  performing  the  assessment  task.  Also,  the  PMA  was  used  to  record 
damage  information,  repairs,  and  for  communications.  Information  provided  via 
the  PMA  included  those  TOs  required  to  successfully  perform  an  ABDR 
assessment.  These  included  the  general  and  F-15  specific  -39  TOs,  TOs  for 
aircraft  structures,  and  TOs  for  relevant  aircraft  systems.  Two  PMAs  were  used 
during  the  field  test,  one  by  the  subject  (client  PMA)  and  one  by  the  field  test 
administrator  (for  communication  and  administration).  One  PMA  acted  as  the 
client  portion  of  the  three-tier  architecture.  The  PMA  provided  information  in 
either  the  CDM  format  or  the  IPDF  format. 

b.  Server  -  a  server  computer  system  used  as  the  information  repository  for  the 
ABDAR  Demonstration  System  databases.  The  ABDAR  Server  executed  the 
database  server  software  and  the  application  server  software.  The  ABDAR 
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Server  was  used  for  storing  and  providing  access  to  the  IPDF  files,  CDM  data, 
and  ABDAR  System  data. 

c.  Administration  Workstation  -  a  desktop  computer  system  used  by  ABDAR  team 
software  engineers  responsible  for  maintaining  the  ABDAR  Demonstration 
System.  The  workstation  was  used  by  software  engineers  to  perform  backups 
and  to  maintain  system  configuration.  Support  personnel  responsible  for 
conducting  the  field  test  (electronic  communication  and  field  test  documentation) 
also  used  this  device. 

For  the  field  test,  the  ABDAR  Demonstration  System  required  the  hardware  and 
software  shown  in  Table  1.  All  of  the  devices  were  configured  with  radio  frequency 
(RF)  modems,  providing  for  wireless  communication.  Additionally,  10-Base  T  Ethernet 
connections  were  used  in  all  devices,  allowing  for  rapid  data  transfers,  communication 
to  home  station,  and  backup  for  the  wireless  communication.  The  ABDAR  server  had  a 
24-hour  service  agreement  with  the  vendor  (Dell  Computer  Corporation),  full  backup 
capabilities,  and  an  uninterruptible  power  supply  (UPS)  ensuring  complete  support  for 
the  field  test. 


Table  1  -  ABDAR  Field  Test  Hardware  and  Software  Configuration 


Equipment 

Configuration 

Requirements 

.Applications/Accessories 

ABDAR  PMA 

5 

(2  for  sub¬ 
ject  use  in 
test  and  3 
for  back-up) 

256MB  RAM 

6  GB  HD 

Win  95/NT  with  Pentium 
class  processor 

Java  Runtime  Environment  1.1.7a 

Adobe  Acrobat  Exchange  3.01  (for  IPDF 
viewing) 

ABDAR  Java  components 

ABDAR  Server 

(Running 

application 

server,  and 

database 

servers) 

1 

256MB  RAM 

5  Hard  Drives  configured 
with  RAID  Level  2  (4  GB 
each) 

Win  NT  with  more  than 
one  Pentium  Pro  or 
Pentium  II  processors 

Application  Server  (Tengah  3.1) 

Database  Server  (Oracle  Enterprise 

Server  7.3) 

ABDAR  Java  components 

ABDAR  Admin 
Workstation 
(Test  Team 
Support) 

1 

256MB  RAM 

6  GB  HD 

Win  95/NT  with  Pentium 
class  processor 

Java  Runtime  Environment  1.1.7a 

Microsoft  Office  97  Suite 

IPDF  Viewing  Capabilities  (Adobe 

Acrobat  Exchange  3.01) 

Database  Management  Capabilities 
(Oracle  Enterprise  Manager) 

Application  Management  Capabilities 
(Tengah  Manager) 

ABDAR  Java  components 
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FIELD  TEST 


The  ABDAR  field  test  was  conducted  in  a  simulated  ABDR  environment.  The  test 
scenario(s)  were  developed  to  represent  a  typical  battle  damage  assessment  situation. 
Data  was  collected  and  analyzed,  with  conclusions  being  drawn  based  on  the  following 
objectives.  Additionally,  this  section  outlines  the  field  test  rules,  scenarios,  and  data 
collection  considerations. 


Objectives 

The  objectives  of  the  ABDAR  field  test  were  to: 

1 .  Collect  data  to  support  statistical  conclusions  regarding  the  following  hypotheses. 

a.  The  F-15  Assessor  using  CDM  data  with  ABDAR  will  perform  significantly 
better  (fewer  errors,  less  time,  more  complete  assessment)  than  the  F-15 
Assessor  using  paper.  . 

b.  The  Other  Assessor  using  CDM  data  with  ABDAR  will  perform  significantly 
better  (fewer  errors,  less  time,  more  complete  assessment)  than  the  F-15 
Assessor  using  paper. 

c.  There  will  be  a  significant  performance  difference  (fewer  errors,  less  time, 
more  complete  assessment)  among  technicians  when  using  CDM  data  with 
ABDAR. 

d.  There  will  be  a  significant  performance  difference  (fewer  errors,  less  time, 
more  complete  assessment)  among  technicians  when  using  I  PDF  data  with 
ABDAR. 

e.  The  F-1 5  Assessor  will  perform  significantly  better  (fewer  errors,  less  time, 
more  complete  assessment)  using  CDM  data  with  ABDAR  than  when  using 
IPDF  data  with  ABDAR. 

f.  There  will  be  a  significant  interaction  between  technician  type  and  media  on 
both  errors  and  time.  CDM  data  with  ABDAR  will  tend  to  minimize  the 
differences  between  technician  types  while  the  Paper  media  with  ABDAR  will 
show  large  differences  between  technician  types.  The  interaction  will  be 
ordinal  in  nature,  meaning  the  relative  ranking  of  the  technician  types  will 
remain  the  same  at  all  three  levels  of  media. 

2.  To  collect  user  feedback  data  on  the  system,  including  recommendations  for 
improvements. 
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Field  Test  Experimental  Design  Considerations 

The  original  field  test  plan  provided  for  a  two  way  ANOVA  with  three  treatment 
variables  (CDM,  IPDF,  Paper)  and  three  categories  of  subjects  (F-15  ABDR  assessors, 
ABDAR  assessors  qualified  on  other  aircraft,  and  F-15  mechanics).  The  design 
required  that  all  subjects  be  qualified  at  skill  level-7  (7-level)  in  their  Air  Force 
Specialties.  However,  a  sufficient  number  of  7-level  F-15  mechanics  were  not 
available,  requiring  the  substitution  of  5-level  mechanics.  Since  the  relative  skill  levels 
of  the  F-15  mechanic  subjects  was  no  longer  comparable  to  the  skill  levels  of  the  F-15 
and  other  aircraft  assessors,  it  was  not  possible  to  directly  compare  their  performance 
with  the  performance  of  the  F-15  assessors  and  other  aircraft  assessors.  A  separate 
analysis  was  performed  on  the  F-15  mechanic  data.  This  change  had  the  effect  of 
breaking  the  field  test  into  two  experiments. 

The  first  experiment  (F-15  and  other  aircraft  assessors)  was  a  2  x  3  factorial  design 
with  technician  type  (F-15  Assessor  and  Other  Assessor)  and  media  type  (Paper,  IPDF, 
and  CDM)  as  the  independent  variables.  Dependent  variables  were  Time,  Accuracy, 
and  Completeness.  Separate  Analysis  of  Variance  (ANOVA)  tests  were  performed  on 
each  dependent  variable.  Subjects  for  the  first  study  consisted  of  30  USAF 
maintenance  Air  Force  Specialty  (AFS)  7-level  qualified  individuals.  Fifteen  were 
ABDAR  qualified  F-15  assessors  and  15  were  ABDAR  qualified  assessors  on  other 
aircraft  (e.g.,  C-130,  F-16). 

The  second  experiment  was  a  1  x  3  design  with  F-15  Mechanics  performing  the 
same  tasks  as  above  with  the  three  media  types  (Paper,  IPDF,  and  CDM).  Dependent 
variables  were  time,  accuracy,  and  completeness.  Separate  One-Way  Analysis  of 
Variance  (ANOVA)  tests  were  performed  on  each  dependent  variable.  Subjects 
consisted  of  18  USAF  maintenance  AFS  5  and  7-level  qualified  individuals.  The  results 
of  this  experiment  are  summarized  in  Appendix  A. 

After  the  field  test  was  completed,  additional  data  was  gathered  to  address  a 
question  raised  in  the  analysis  of  the  field  test  data.  The  additional  data  was  collected 
to  address  the  issue  of  whether  performance  advantages  observed  for  the  ABDAR 
IPDF  data  were  due  to  the  implementation  of  IPDF  data  in  the  ABDAR  system  or  due  to 
the  fact  that  the  data  is  presented  electronically.  In  other  words,  would  the  same 
performance  advantages  have  been  observed  if  the  IPDF  data  had  been  presented  in 
its  basic  form  without  the  enhancements  and  aids  provided  by  the  ABDAR  System? 
The  findings  of  this  additional  data  collection  are  presented  in  Appendix  B. 

Subjects 

For  both  experiments,  subject  assignment  was  randomized  to  minimize  spurious 
effects  of  order,  shift,  experimenter,  and  other  non-experimental  variables.  For 
randomized  subject  assignments  see  APPENDIX  C  -  FIELD  TEST  SCHEDULE,  Table 
C-  1  -  Paper  Schedule,  Table  C-  2  -  Electronic  One  Schedule,  and  Table  C-  3  - 
Electronic  Two  Schedule. 
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A  total  of  Fifty-three  subjects  participated  in  the  two  experiments.  All  subjects  were 
identified  before  the  start  of  the  experiment.  Every  effort  was  made  to  avoid 
substitutions.  One  F-15  assessor  was  unable  to  complete  the  testing,  reducing  the  total 
number  of  subjects  to  53  and  the  number  of  subjects  used  in  the  main  analysis  to  30^ 
for  Experiment  One  and  18  subjects  for  Experiment  Two. 

The  subjects  for  the  first  experiment  were  fifteen  ABDR  qualified  F-15  assessors  (4 
crew  chiefs,  2  fuel  specialists,  4  electrical  specialists,  and  5  structural  specialists)  and 
fifteen  ABDR  assessors  qualified  on  other  aircraft  (7  crew  chiefs,  1  hydraulic  specialist, 
1  electrical  specialist,  and  9  structural  specialists).  All  were  qualified  at  skill  level-7  in 
their  specialties. 

The  subjects  for  the  second  experiment  were  eighteen  qualified  F-15  mechanics  (10 
crew  chiefs  and  8  structural  specialists).  The  subjects  included  both  5-level  and  7-level 
technicians. 

ABDAR  Data  Collection  Team  Composition  (Paper  and  ETI) 

Based  upon  scheduling  considerations  and  the  variety  of  skills  required,  two  data 
collection  teams  were  formed  to  facilitate  a  two-shift  operation. 

a.  Positions  and  Responsibilities.  The  requirements  for  the  field  test  team  with 
positions  and  responsibilities  are  provided  below: 

1 .  Field  Test  Director.  The  field  test  director  position  was  the  focal  point  for  the 
team.  This  individual  understood  the  objectives,  and  the  pitfalls,  associated 
with  doing  the  test.  It  was  the  responsibility  of  the  field  test  director  to  ensure 
all  personnel,  equipment,  subjects,  and  other  requirements  were  available 
when  needed.  Also,  the  test  director  was  responsible  for  ensuring 
consistency  in  the  testing  process  throughout  the  field  test. 

2.  Team  Leader.  Two  individuals  filled  the  position  of  team  leader  due  to 
multiple  shift  operations.  The  team  leaders  assisted  the  director  in  his  efforts, 
were  the  focal  point  for  each  team,  and  monitored  the  day-to-day  operations 
of  the  team.  The  team  leader  also  fulfilled  the  Trainer-Briefer  role.  This 
included  performing  the  in-briefing,  training,  and  out-briefing  of  the  subjects. 
Subjects  required  approximately  two  hours  of  training.  The  team  leader  also 
ensured  that  the  subject  understood  what  was  desired  when  documenting  the 
assessment,  how  to  determine  completion  of  ABDR,  and  the  rules  to  be 
followed  during  the  test. 


^  The  experimental  design  required  that  the  number  of  observations  in  each  cell  be  equal.  Since  one  F-15 
assessor  was  not  able  to  complete  the  testing,  it  was  necessary  to  drop  one  observation  from  each  of  the 
other  cells  to  give  the  same  number  of  observations  per  cell.  The  F-15  assessor  who  could  not  participate 
had  been  designated  as  subject  #5  in  the  F-15  assessor  group.  To  equalize  the  number  of  subjects  per 
group,  the  data  for  subject  #5  for  each  assessor  group  was  dropped  from  the  analysis. 
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(a)  In-briefing  of  Subjects.  The  following  are  the  areas  that  were  discussed 
with  the  test  subjects  before  the  start  of  the  test. 

(1)  An  overview  of  the  ABDAR  technology  effort. 

(2)  Work  schedules. 

(3)  Rules  of  the  test. 

(4)  Expectations  of  the  subjects. 

(5)  Emergency  telephone  numbers  and  contacts. 

(6)  Consent  form,  provided  byAFRL/HESR. 

(7)  Demographic  information  form  (APPENDIX  D). 

(b)  Out-briefing  of  Subjects.  The  following  are  the  areas  that  were  discussed 
with  the  test  subjects  before  releasing  them  back  to  their  owning 
organizations. 

(1)  Exit  Questionnaire  (APPENDIX  E)  for  ETI  subjects  only. 

(2)  Confidentiality  of  test  specifics. 

(3)  Appreciation  for  their  effort. 

3.  Test  Monitor/SME.  The  test  monitors  were  responsible  for  monitoring  the 
subjects  performance  and  collecting  all  performance  data.  The  test  monitors 
also  served  as  subject  matter  experts  for  the  test. 

(a)  The  SME’s  responsibilities  included: 

(1)  Providing  maintenance  expertise  on  the  tasks  being  accomplished  on 
the  aircraft. 

(2)  Extensive  knowledge  of  the  ABDAR  Demonstration  System  and  its 
technical  data. 

(3)  Assisting  the  subject  when  help  was  required. 

(4)  Collecting  data. 

(5)  Assisting  in  tool  and  equipment  accountability. 

(6)  identifying  and  documenting  errors  when  they  occurred  during  the 
session. 

(7)  Recording  time  measurements. 
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(8)  Scoring  the  data  collection  sheets  after  each  test. 

(b)  The  SME’s  daily  tasks  included: 

(1)  Preparing  the  aircraft  before  the  arrival  of  the  subject. 

(2)  Acting  as  the  Safety  Officer. 

(3)  Calling  time  outs  to  recover  the  ABDAR  Demonstration  System  when 
bugs  occurred. 

(c)  Other  roles  the  SME  personnel  fulfilled  were  the  ABDAR  team  positions  of 
Resource  Manager,  Pro  Super,  Engineer,  ABDR  Technician,  and  Team 
Chief.  These  individuals,  familiar  with  the  ABDR  process  and  the  ABDAR 
Demonstration  System,  were  always  available  and  provided  consistent 
information  and  feedback  to  the  assessor.  Due  to  the  large  amount  of 
interaction  between  this  position  and  the  assessor,  in  the  electronic 
version,  this  function  required  full  time  attention. 

(d)  Support  Personnel.  The  Support  Personnel  were  only  needed  for  the 
electronic  portions  of  the  field  test.  They  were  available  for  each  shift  and 
maintained  the  database,  server,  and  PMA  software.  Additionally,  support 
personnel  provided  support  for  the  LAN  and  hardware. 


Facilities 

Figure  2  provides  a  graphical  presentation  of  the  layout  of  the  hangar  facility  located 
at  Robins  AFB,  GA.  The  use  of  an  office,  within  sight  of  the  planned  aircraft  parking 
location,  was  required.  This  office  was  used  as  the  team  headquarters,  briefing  and 
training  facility,  and  housed  the  workstation  and  ABDAR  Field  Test  server.  The  office 
had  air-conditioning,  electrical  service,  and  telephone  and  base  Local  Area  Network 
(LAN)  hook-ups  to  support  the  workstation  and  ABDAR  Field  Test  server.  The  office 
could  be  locked,  providing  a  secure  area  for  storage  of  PMAs,  battery  chargers,  and 
other  miscellaneous  equipment  and  materials.  The  Radio  Frequency  (RF)  LAN 
antenna  was  mounted  on  top  of  another  office  and  in  line  of  sight  of  the  aircraft.  Along 
with  the  dedicated  aircraft  and  hangar  space,  the  following  items  were  provided  for  the 
subjects  at  the  aircraft. 
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CL5S/ABDR  Offices 


Figure  2  -  Robins  AFB  Hangar  Facility  Layout 


a.  Worktable  and  chairs. 

b.  1 0-foot  ladder. 

c.  Assessor  Tool  Kit. 

d.  A  TO  library  containing  a  sufficient  selection  of  TOs  to  cover  the  damages  for 
use  by  subjects  performing  the  assessments  using  paper  technical  data  and  as  a 
reference  for  the  team  members  (paper  technical  order  condition  only). 
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Damage  Information 


In  determining  the  areas  of  the  aircraft  that  were  to  be  damaged  for  the  field  test, 
three  basic  criteria  were  used. 

a.  The  damaged  areas  must  include  functional  and  structural  components  to  be 
representative  of  most  types  of  assessment  performed  in  ABDR.  Therefore, 
areas  were  chosen  that  contained  the  following  types  of  functional  components: 
avionics,  pneumatic,  hydraulic,  mechanical  flight  control,  electrical,  and  wire 
bundles.  This  functional  breakdown  is  identical  to  the  breakdown  found  in  the  1- 
1H-39  and  AFTO  Form  97.  Additionally,  the  damaged  areas  were  required  to 
contain  varied  types  of  structural  components.  The  damaged  areas  were  to 
include  all  five  categories  of  structural  components  and  to  represent  both  types 
of  structural  assessment  (K-factor  and  class)  used  in  ABDR  on  F-1 5s. 

b.  The  location  and  extent  of  the  damages  must  be  consistent  with  test  constraints 
(time,  safety,  security,  etc.).  The  ease  of  conducting  and  administering  the  field 
test  was  taken  into  consideration  when  determining  damage  location.  To 
maximize  the  use  of  the  aircraft,  it  was  prudent  to  damage  areas  of  the  aircraft 
that  were  not  close  to  each  other.  To  minimize  the  workload  of  the  subjects, 
each  damage  must  be  able  to  be  assessed  during  a  single  working  session. 

Damages  must  be  representative  of  damages  incurred  during  combat  operations.  The 
procedures  for  simulating  battle-damage,  as  described  in  the  1-1H-39,  were  used  to 
inflict  damage  on  the  test  sites.  The  process  produced  damages  that  realistically 
simulate  damages  inflicted  in  combat. 

Two  damage  sites  were  created.  Each  served  as  a  separate  test  problem  for  the 
evaluation. 

The  damaged  areas  did  not  require  use  of  classified  data  and  did  not  create  any 
major  safety  problems,  other  than  height  (approximately  7.6  ft.  at  wing  tip). 
Concentrating  on  these  areas  enabled  the  testing  of  a  wide  range  of  functional 
assessments  and  minimized  the  need  to  develop  large  volumes  of  redundant  technical 
data.  The  following  areas  were  damaged  for  the  field  test. 

a.  Door/Bay  6R  in  Zone  4.  This  area  contains  18  different  functional  components, 
25  LRUs,  approximately  30  wire  bundles,  and  approximately  50  structural 
components.  The  functional  systems  included  electrical,  avionics,  and 
miscellaneous  (e.g.,  LOX  and  environmental  cooling).  The  approximately  50  1F- 
15A-39-indexed  Category  I,  II,  IV,  and  V  structures  used  the  F-15  K-factor 
evaluation  method  for  determining  serviceability.  The  damage  was  inflicted 
using  a  pickaxe  and  screwdriver,  simulating  a  close  proximity  explosion.  The 
damage  infliction  process  resulted  in  14  damaged  components. 

b.  Left  Wing  Trailing  edge  in  Zone  11.  This  area  included  6  Systems,  31  LRUs,  3 
wire  bundles,  and  approximately  65  structures.  The  functional  systems  included 
hydraulics,  pneumatics,  and  mechanical  flight  controls.  The  structural 
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components  were  category  I,  II,  and  V,  and  used  diameter  and  location  as  the 
evaluation  method.  The  damage  was  inflicted  using  C4  explosives,  simulating 
penetration  of  a  projectile.  The  damage  infliction  process  resulted  in  8  damaged 
components. 

Data  Collection  Sheet 

The  data  collection  sheets  (APPENDIX  F)  were  designed  to  help  the  data  collector 
track  and  record  information  on  the  technician’s  performance.  The  sheets  provided  an 
outline  of  the  task,  identified  specific  subtasks,  and  provided  places  to  record  the  start 
and  stop  times  for  each  subtask.  Cues  and  reminders  were  provided  to  help  ensure  the 
data  collector  did  not  overlook  essential  points  or  actions.  The  data  collection  sheet 
provided: 

a.  “Cheat  sheet,”  type  information  for  the  monitors,  organized  to  support  ease  of 
data  collection. 

b.  Places  to  record  start  and  stop  times 

c.  Identification  of  which  electronic  documents  would  be  needed  for  evaluation  and 
analysis  later. 

d.  A  place  to  record  which  damages  were  found  and  which  were  pointed  out  to  the 
subject.^ 

e.  A  place  for  documentation  of  any  errors  identified,  along  with  documentation  of 
the  solution. 


Field  Test  Rules 

Test  rules,  APPENDIX  G  and  APPENDIX  H,  were  established  for  the  subjects  and 
all  members  of  the  test  team.  Subject  rules  were  established  so  the  subjects  had  a 
clear  understanding  of  what  was  expected  of  them.  The  rules  were  briefed  as  part  of 
the  in-briefing  and  were  available  throughout  the  field  test.  Test  Monitor  rules  provided 
the  ground  rules  that  were  established  for  the  monitors  and  other  positions  of  the  data 
collection  team  to  ensure  “consistent  and  equal  treatment”  of  subjects  throughout  the 
field  test. 


^  Damages  that  the  subject  did  not  find  within  a  specified  time  were  pointed  out,  so  that  the  subject  could 
plan/specify  repairs  for  all  damages.  This  was  essential  to  ensure  that  all  subjects  developed  planned 
repairs  for  the  same  set  of  damages. 


13 


Field  Test  Scenario  (Paper  and  ETI’s) 

Since  the  primary  focus  of  the  ABDAR  Demonstration  System  was  to  improve  the 
speed,  accuracy,  and  completeness  of  assessment,  the  field  test  targeted  the 
assessment  portion  of  the  ABDR  process.  The  assessment  portion  of  the  ABDR 
process  consists  of  Damage  Collection,  Repair  Selection,  Repair  Planning,  and  the 
Documentation  associated  with  each  of  these  processes.  Each  session  started  with  a 
completed  Debrief  and  ended  with  Selected  Repairs  planned  for  accomplishment.  The 
need  for  interaction  with  Status  Boards  and  Resource  Management  was  based  on  the 
situation  at  any  given  time.  Standard  Monitor/SME  responses  were  developed  for  all 
anticipated  interactions  and  questions  from  the  subjects,  so  that  all  requests  and 
questions  were  handled  identically.  In  those  instances  where  the  interaction  was 
novel,  the  test  monitor(s)  created  and  provided  logical  answers  on  the  spot.  The 
created  answers  became  part  of  the  available  set  of  interaction  responses  for  use  if  the 
need  arose  again. 

Test  scenarios  were  performed  on  two  damage  sites  and  graded  according  to  the 
criterion  identified  in 

Evaluations  and  Scoring  (page  15).  Each  damage  site  was  treated  as  a  separate 
test  problem  for  analysis  purposes. 

Subjects  using  Paper  received  scenarios  provided  in  APPENDIX  I  and  APPENDIX 
J.  Subjects  using  the  electronic  media  received  scenarios  shown  in  APPENDIX  K  and 
APPENDIX  L.  The  scenarios  are  identical  except  for  the  type  of  technical  data  used. 


Field  Test  Data  Collection  Considerations 


Test  Problems 

Two  separate  sections  of  the  aircraft  were  damaged  for  the  field  test.  Each  damage 
site  was  treated  as  a  separate  problem.  The  Avionics  Bay  damage  site  had  14 
damages.  The  Wing  damage  site  had  eight.  A  complete,  accurate  assessment, 
according  to  TO  1-1H-39,  includes  a  detailed  description  of  each  damage  and  the 
chosen  repair  action.  TO  1-1H-39  defines  the  minimum  requirements  for  describing  a 
damage  and  a  repair. 

As  stated  in  TO  1-1H-39,  each  damage  requires  a  clear  and  complete  description  to 
ensure  that  another  assessor  can  pick  up  and  continue  the  assessment,  without  having 
to  reassess  any  damage(s).  To  meet  this  goal,  the  assessor  must  identify  the  entities 
that  are  damaged,  give  the  exact  location  of  the  damage,  and  describe  the  extent  of  the 
damage.  The  specific  -39  has  an  indexed  graphic  with  a  corresponding  table  of 
information  about  each  entity.  Information  from  this  table  is  useful  to  describe  the 
damage,  its  limits,  and  the  actions  to  take  for  evaluation  and  repair  of  the  damage. 
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Evaluations  and  Scoring 

To  assist  in  maintaining  objectivity  and  accuracy,  assessments  were  evaluated 
immediately  after  completion.  The  test  monitors  evaluated  the  assessment  information 
documented  by  the  subject  using  the  criteria  shown  in  APPENDIX  M.  A  score  was 
then  assigned  in  accordance  with  APPENDIX  N  and  APPENDIX  O.  The  information 
collected  by  the  test  monitors  was  tabulated  and  attached  to  the  evaluation  data  packet 
for  analysis.  The  following  areas  were  evaluated  and  scored. 

a.  Structure  Damage  Collection 

b.  System  Damage  Collection 

c.  Wiring  Damage  Collection 

d.  Repair  Selection/Instructions 

e.  Repair  Planning  and  Scheduling 

f.  Other  Areas 

(1)  UXO  Inspection  and  documentation 

(2)  Use  and  documentation  of  Engineering  Input 

(3)  Estimated  Times 

(4)  Highest  Category  and  Class  Documentation 

(5)  Number  of  Found  Damages 


FIELD  TEST  SCHEDULE 

Ideally,  it  would  have  been  preferable  to  completely  randomize  the  presentation  of 
the  three  types  of  media,  however,  real-world  commitments  and  schedules  did  not 
permit  this  approach.  Therefore,  the  decision  was  made  to  run  subjects  using  Paper 
media  in  the  first  three-vveek  experimental  block,  subjects  using  CDM  in  the  second, 
and  subjects  using  IPDF  in  the  third.  Subjects  in  the  Paper  media  condition  were  run  in 
the  fall  of  1998  and  subjects  using  the  CDM  and  IPDF  media  were  run  in  the  fall  of 
1999.  The  following  assumptions  were  made  when  creating  the  field  test  schedule: 

a.  Estimated  six  hours  per  session. 

b.  Only  one  active  session  at  a  time. 
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c.  One  Monitor/SME  would  observe  a  subject  as  the  subject  performed  the 
assessment  task.  Monitor/SMEs  would  be  allowed  to  switch  within  a  session  or 
between  sessions. 

d.  In-brief,  training,  and  out-brief  accomplished  by  someone  other  than  the  test 
monitor  (team  leader). 

e.  Two  data  collection  shifts  per  day. 

The  field  test  period  was  divided  into  three  three-week  testing  blocks.  This  breakout 
supported  54  subjects,  which  equates  to  18  subjects  per  media  variable.  With  six 
subjects  per  week,  each  three-week  block  supported  18  subjects.  See  APPENDIX  C  - 
FIELD  TEST  SCHEDULE,  Table  C-  1  -  Paper  Schedule,  Table  C-  2  -  Electronic  One 
Schedule,  and  Table  C-  3  -  Electronic  Two  Schedule. 

Each  subject  accomplished  the  required  training  prior  to  the  first  session.  The 
subjects  then  participated  in  two  experimental  sessions,  followed  by  an  out-briefing  at 
the  end  of  the  last  session.  The  schedule  required  two  full  days  of  a  subject’s  time. 
This  did  not  include  travel  time  for  those  subjects  that  were  assigned  on  temporary  duty 
(TDY)  to  the  test  site.  On  the  first  day,  the  subjects  trained  and  participated  in  the  first 
experimental  session.  The  second  day  the  subjects  completed  the  second 
experimental  session  and  the  out-brief  from  the  experiment. 


TRAINING  CONSIDERATIONS 

Training  issues  for  the  ABDAR  field  test  centered  on  the  training  of  the  data 
collection  team  and  the  training  of  the  subjects.  Subjects  received  equivalent  training, 
regardless  of  the  media  type  being  used.  Training  of  the  data  collection  teams  was 
designed  and  validated  during  the  Pre-Field  Test. 


Subjects 

All  subjects,  regardless  of  the  media  condition  used  (Paper,  IPDF,  or  CDM), 
received  training  before  the  ABDAR  field  test.  Subjects  trained  only  on  the  media  that 
they  used  during  the  field  test.  Training  allowed  the  subjects  to  become  familiar  with 
the  media  condition  they  would  be  using. 

Tutorials  were  developed  for  subjects  in  the  CDM  and  IPDF  conditions.  Subjects  in 
the  paper  condition  used  the  tutorials  contained  in  TO  1-1H-39  and  TO  1F-15A-39. 
Subjects  trained  themselves  on  ABDR  using  their  designated  media  as  the  training 
technique.  The  subjects  in  the  Paper  media  condition  used  the  TO  1-1H-39  tutorial  and 
other  technical  manuals  to  teach  themselves  ABDR.  The  subjects  in  the  IPDF  media 
condition  trained  themselves  in  ABDR  using  the  paper  tutorial  and  the  ABDAR 
Demonstration  System.  The  paper  tutorial  provided  instructions  on  how  to  find  the 
required  information  on  the  computer  and  provided  opportunities  to  practice  data 
retrieval  on  the  system.  The  subjects  using  the  CDM  media  trained  themselves  on  the 
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ABDAR  Demonstration  System  with  a  similar  paper  tutorial,  which  presented  the  basic 
procedures  for  using  the  electronic  CDM  version. 

In  all  conditions,  the  Trainer-Briefer  was  present  to  assist  the  subject  and  answer 
questions.  The  goals  of  the  self-paced  training  sessions  were  provided  to  the  subjects 
as  a  self-measuring  device.  To  successfully  complete  the  training,  the  subject  must  be 
able  to: 

a.  Use  the  media  to  locate  and  evaluate  a  damage. 

b.  Use  the  media  to  identify  repair  options. 

c.  Document  a  damage  assessment  according  to  specifications  established  by  TO 
1-1H-39. 

d.  Identify,  acquire,  and  evaluate  real  time  information  needed  to  make  repair 
decisions. 

e.  Plan  repairs  making  effective  use  of  time  and  resources. 


Data  Collection  Team  Position  Training 

Position  Training  took  place  during  the  Pre-Field  Test.  Training  for  the  test  monitors 
included  directions  for  documenting  the  data  collection  sheets,  preparation  of  the 
aircraft  before  each  session,  interfacing  with  the  test  subject,  and  safety  considerations. 
Training  for  the  team  leaders  included  how  to  in-brief,  out-brief,  use  the  training 
materials,  scheduling  and  coordinating  subjects,  and  general  management  of  the  field 
test. 


RESULTS  OF  THE  ABDAR  FIELD  TEST 

The  data  for  each  of  the  main  dependent  variables  were  subjected  to  an  Analysis  of 
Variance  (ANOVA)  test.  Data  for  the  two  testing  sessions  (damage  site  one  and 
damage  site  two)  were  combined  to  provide  a  single  score  on  each  subject  on  each 
measure  (Time,  Accuracy,  and  Completeness).  The  ANOVA’s  for  Time, 
Completeness,  and  Accuracy  were  planned  a  priori.  Additional  post  hoc  testing  was 
performed  according  to  trends  in  the  data.  If  an  ANOVA  was  significant  for  a  variable,  a 
Bonferroni  Multiple  Comparison  test  was  used  to  determine  statistical  significance  of 
differences  between  the  means.  For  example,  the  ANOVA  for  Time  was  significant  for 
the  independent  variable  of  Media.  Therefore,  there  was  a  difference  between  the 
mean  scores  for  the  fastest  variable  (subjects  using  CDM)  and  the  slowest  variable 
(subjects  using  IPDF).  The  ANOVA  does  not  supply  any  information  about 
relationships  with  the  intermediate  variable  (subjects  using  Paper).  The  Bonferroni 
Multiple  Comparison  Test  allowed  determination  of  significance  between  any  two  of  the 
means. 
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The  Time  variable  was  comprised  of  the  amount  of  time  a  subject  took  to  complete 
the  assessment,  including  identification  of  damage,  evaluation  of  damage  and  repair 
planning.  The  monitors  recorded  (in  minutes)  the  time  that  each  subject  worked  on  the 
assessment  task,  not  including  breaks  and  time  outs  for  unavoidable  interruptions  (such 
as,  system  malfunctions). 

The  Completeness  and  Accuracy  Variables  were  derived  from  the  “perfect 
assessment”,  developed  before  the  field  test  and  data  collection.  To  develop  the 
“perfect  assessment”  three  highly  experienced  ABDR  assessors  were  employed.  Each 
assessor  was  also  an  experienced  instructor,  adding  to  the  credibility  and  expertise  of 
the  three.  Each  assessor  performed  the  assessment  individually.  As  was  expected, 
the  three  assessments  differed.  The  test  monitors  then  lead  the  assessors  back 
through  all  of  the  assessments,  pointing  out  discrepancies.  From  this  exercise,  the 
three  assessors  came  to  a  consensus,  a  “perfect  assessment”  which  was  used  as  the 
standard  for  grading  the  Accuracy  and  Completeness  variables. 

The  Completeness  Variable  reflected  the  thoroughness  of  an  assessment.  If  the 
perfect  assessment  had  100  documentation  requirements  annotated  and  the 
experimental  assessment  had  76  documentation  requirements  annotated,  the 
assessment  would  receive  a  score  of  76%  complete  (76  annotated/100  total). 

The  Accuracy  Variable  reflected  how  many  documentation  requirements  were 
annotated  correctly  in  an  experimental  assessment.  To  continue  the  example  from 
above,  suppose  that  of  the  76  documentation  requirements  annotated,  only  56  were 
annotated  identically  to  the  perfect  assessment.  In  this  case,  the  experimental 
assessment  would  receive  a  score  of  56%  Accurate  (56  correct  annotations/ 100  total). 
Why  not  a  score  of  74%  Accurate  (56  accurately  annotated/76  annotated)?  It  was 
decided  that  only  considering  the  accuracy  of  those  damages  that  were  annotated  was 
not  as  precise  a  measure  as  the  accuracy  of  all  possible  documentation  requirements. 
When  determining  accuracy,  using  only  the  annotated  documentation  requirements, 
instead  of  the  documentation  requirements  from  the  perfect  assessment,  is  more 
reflective  of  the  completeness  of  an  assessment  than  the  accuracy.  Simply  put,  if  a 
documentation  requirement  was  missed  during  assessment,  it  was  not  accurately 
assessed. 


Statistical  Analysis 

Results  of  the  statistical  analysis  for  each  variable  are  provided  in  the  following 
paragraphs. 

Time 

The  ANOVA  for  the  dependent  variable  of  Time  (Table  2)  showed  a  significant 
effect  of  Media,  F  (2,29)  =  15.58,  p  <  .0000.  Assessors  using  CDM  performed  quickest 

{x  =  337.6  minutes),  followed  by  assessors  using  Paper  (x  =  447.9),  and  assessors 
using  IPDF  (x  =  629.6).  Bonferroni  Multiple  Comparison  tests  show  significant 
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differences  between  the  means  of  subjects  using  CDM  and  subjects  using  IPDF  and 
between  the  means  of  subjects  using  Paper  and  subjects  using  CDM. 

Table  2  -  Analysis  of  Variance  Table  -  Time 


Source 

Sum  of 

Mean 

Prob 

Term 

DF 

Squares 

Square 

F-Ratio 

Level 

A:  Assessor 

1 

32.03333 

32.03333 

0.00 

0.962186 

B:  Media 

2 

434816.6 

217408.3 

15.58 

0.000046* 

AB 

2 

22821.27 

11410.63 

0.82 

0.453426 

S 

24 

334962.4 

13956.77 

Total  (Adjusted) 

29 

792632.3 

Total 

30 

‘Term  significant  at  alpha  =  0.05 

There  was  no  significant  effect  for  Technician  Type,  nor  was  there  an  interaction 
between  Technician  Type  and  Media. 


Completeness 

The  ANOVA  for  the  dependent  variable  of  Completeness  (Table  3)  showed  a 
significant  effect  of  Media,  F  (2,29)  =  82.47,  p  <  .0000.  Assessors  using  CDM 
completed  95.4%,  IPDF  Assessors  completed  90.3%,  and  Assessors  using  Paper 
completed  56.8%  of  the  perfect  assessment.  A  Bonferroni  Multiple  Comparison  Test 
showed  assessors  using  CDM  and  IPDF  completed  significantly  more  of  the  perfect 
assessment  than  subjects  using  Paper.  There  was  not  a  significant  difference  between 
subjects  using  CDM  and  IPDF. 


Table  3  -  Analysis  of  Variance  Table  -  Completeness 


Source 

Sum  of 

Mean 

Prob 

Term 

DF 

Squares 

Square 

F-Ratio 

Level 

A:  Assessor 

1 

197.6333 

197.6333 

3.71 

0.066121 

B;  Media 

2 

8794.066 

4397.033 

82.47 

0.000000“ 

AB 

2 

36.86666 

18.43333 

0.35 

0.711168 

S 

23 

1279.6 

53.31667 

Total  (Adjusted) 

29 

10308.17 

Total 

30 

‘Term  significant  at  alpha  =  0.05 
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There  was  no  significant  effect  for  Technician  Type.  There  was  no  interaction 
between  Technician  Type  and  Media 


Accuracy 

The  ANOVA  for  the  dependent  variable  of  Accuracy  (Table  4)  showed  a  significant 
effect  of  Media,  F  (2,29)  =  40.08.  p  <  .0000.  Assessors  using  CDM  achieved  an 
accuracy  rate  of  90.9%,  Assessors  using  IPDF  were  83.3%  accurate,  and  Assessors 
using  Paper  were  only  39.8%  accurate.  A  Bonferroni  Multiple  Comparison  Test  showed 
assessors  using  CDM  and  IPDF  to  be  significantly  more  accurate  than  assessors  using 
Paper.  There  was  not  a  significant  difference  between  assessors  using  CDM  and 
IPDF. 


Table  4  -  Analysis  of  Variance  Table  -  Accuracy 


Source 

Term 

DF 

Sum  of 

Squares 

Mean 

Square 

F-Ratio 

Prob 

Level 

A:  Assessor 

1 

282.1333 

282.1333 

3.94 

0.0586 

B:  Media 

2 

15204.07 

7602.033 

0.000000* 

AB 

2 

59.26667 

29.63333 

0.41 

0.665527 

S 

23 

1717.2 

71.55 

Total  (Adjusted) 

29 

17262.67 

Total 

30 

*Term  significant  at  alpha  =  0.05 

There  was  no  significant  effect  for  Technician  Type.  There  was  no  interaction 
between  Technician  Type  and  Media. 


Interpretation  of  Test  Results 

A  major  factor  impacting  the  analysis  of  the  ABDAR  field  test  data  is  the  limited 
number  of  subjects.  Due  to  the  limited  subject  pool,  the  extensive  time  required  to 
collect  data  for  each  subject,  and  scheduling  issues,  only  five  subjects  participated  in 
any  one  condition  of  the  main  study.  This  factor  alone  had  the  greatest  impact  on 
analyzing  the  data.  The  combination  of  limited  subjects  and  a  complex  and  time- 
consuming  task  resulted  in  very  wide  performance  variances  within  a  condition.  The 
consequence  of  the  wide  variances  is  a  lack  of  precision  in  the  results.  The  wide  range 
of  performance  times  limits  the  identification  of  statistically  significant  effects  to  only 
large  effects,  such  as  the  speed  difference  between  subjects  using  CDM  and  IPDF. 

Findings  for  each  of  the  dependent  variables:  Time,  Completeness,  and  Accuracy 
are  discussed  and  interpreted  in  the  following  paragraphs. 
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Time 


Subjects  in  Experiment  One  using  CDM  performed  the  assessment  faster  than 
subjects  using  the  other  two  media  types.  Assessors  using  CDM  finished  nearly  two 
hours  sooner,  on  average,  than  their  counterparts  using  Paper,  and  more  than  five 
hours  faster  than  the  subjects  using  IPDF.  As  mentioned  previously,  the  large  standard 
deviation  associated  with  the  subjects  using  Paper,  almost  2.5  hours  (149.86  minutes), 
prevents  any  statistical  tests  from  identifying  a  potential  difference  between  subjects 
using  CDM  and  Paper.  However,  subjects  using  Paper  performed,  on  average,  181.7 
minutes  quicker  than  subjects  using  IPDF.  Even  with  the  large  standard  deviations 
experienced  under  both  media  conditions,  the  three-hour  difference  is  statistically 
significant.  The  five-hour  difference  between  subjects  using  CDM  and  IPDF  was  also 
significant,  according  to  the  Bonferroni  Multiple  Comparison  Test.  See  Table  5  for  Time 
means,  standard  deviations,  and  ranges  for  all  conditions  for  the  F-15  assessors  and 
other  aircraft  assessors. 


Table  5  -  Mean  Time  (in  minutes)  for 
Assessment  of  Both  Damage  Sites 


CDM 

IPDF 

Paper 

Mean 

337.60 

629.60 

447.90 

Std.  Dev. 

43.90 

123.99 

149.86 

Range 

141 

.  369 

412 

There  is  very  little  clarity  in  the  Time  data.  Figure  3  shows  the  problem  with  the 
Time  data,  namely,  it  has  too  wide  of  a  range  to  permit  many  inferences.  The 
performances  of  subjects  using  CDM  are  obviously  superior  to  subjects  using  IPDF 
because  there  is  only  a  three-minute  overlap  in  their  respective  ranges.  On  the  other 
hand,  the  range  of  times  for  the  subjects  using  Paper  is  so  wide  that  it  nearly 
encompasses  the  full  ranges  of  both  of  the  other  two  media  types.  Due  to  the 
excessive  range  and  huge  variability  associated  with  the  Time  variable,  particularly  with 
the  subjects  using  Paper,  it  is  difficult  to  draw  firm  conclusions  from  the  time  data 
without  consideration  of  the  impact  of  time  on  the  variables  of  completeness  and 
accuracy.  The  Time  data  should  be  viewed  only  in  context  with  the  other  dependent 
variables.  A  fast  assessment  time  is  moot  if  the  assessor  only  completes  50%  of  the 
task.  A  short  assessment  time  is  only  relevant  if  the  assessment  is  also  accurate  and 
complete. 
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Figure  3  -  Box  Plot  for  Time  Data 

(NOTE:  The  graph  shows  the  median,  inter-quartile  range,  and  adjacent  values  for 
each  of  the  media  types  for  the  dependent  variable  of  Time.) 


It  should  be  noted  that  subjects  using  the  IPDF  media  type  always  showed 
improvement  in  damage  assessment  time  between  the  first  and  second  damage  site. 
This  implies  a  strong  learning  curve  associated  with  the  IPDF.  This  has  ramifications 
for  the  interpretation  of  the  time  data.  More  training  with  IPDF  or  simply  more 
experience  will  reduce  the  time  taken  to  perform  an  assessment. 

There  are  several  possible  explanations  for  technicians  using  IPDF  data  taking  so 
much  longer  to  perform  the  assessment  task  than  technicians  using  the  paper  TO.  The 
most  plausible  explanation  is  that  the  ABDAR  Demonstration  System  provides  a 
structure  and  aids  that  guide  the  technician  systematically  through  the  process, 
ensuring  accuracy  and  completeness,  while  the  technician  using  the  paper  TO  is  left  on 
his  own.  In  other  words,  the  ABDAR  IPDF  system  forces  the  technician  to  follow  a 
structured  process  that  insures  a  more  complete  assessment.  The  technician  using 
paper  does  not  have  this  forced  structure  and,  consequently,  is  more  likely  to  miss 
important  information.  This  may  help  to  explain  the  wide  range  of  scores  for  the  paper 
condition.  The  technician  using  Paper  who  by  nature  is  thorough  and  complete  in  his 
work  may  have  taken  much  longer  to  complete  the  assessment,  while  the  technician 
who  tends  to  be  less  thorough  took  much  less  time.  This  interpretation  is  supported  by 
the  fact  that  there  is  a  high  correlation  (r=.88)  between  the  Time  and  Completeness 
scores  for  the  technicians  using  paper,  (compared  to  r=.46  for  IPDF  and  .51  for  CDM). 
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Completeness 

The  Completeness  of  an  assessment  is  the  measure  of  how  thoroughly  the 
assessor  annotated  the  documentation  requirements.  The  documentation  did  not  need 
to  be  annotated  exactly  as  described  in  the  perfect  assessment,  so  long  as  it  was 
recorded  and  the  information  was  correct. 

The  Completeness  data  is  more  robust  than  the  Time  data  and  allows  a  more 
precise  analysis.  As  with  the  Time  variable,  the  only  significant  Main  Effect  was  found 
for  the  Media  variable.  In  this  case,  assessors  using  COM  and  IPDF  performed 
significantly  better  than  the  assessors  using  Paper.  In  fact,  assessors  using  CDM  and 
IPDF  performed  so  well  that  it  is  possible  the  task  was  not  difficult  enough  to  discern 
any  difference  between  subjects  using  the  two  media  types  (Figure  4).  Subjects  using 

the  CDM  and  IPDF  media  types  performed  above  the  90%  complete  level  (x  =  95.4 
and  90.3,  respectively).  A  more  complex  task  might  reveal  a  performance  difference 
between  these  two  media  types.  See  Table  6  for  means,  standard  deviations,  and 
ranges  for  all  conditions.  Figure  4  provides  a  box  plot  for  the  completeness  scores. 
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Figure  4  -  Box  Plot  for  Completeness 

(NOTE;  The  graph  shows  the  median,  inter-quartile  range,  and  adjacent  values  for 
each  of  the  media  types  for  the  dependent  variable  of  Time.) 
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Table  6  -  Descriptive  Statistics  for  Percent  Complete 


CDM 

IPDF 

Paper 

Mean 

95.40 

90.30 

56.80 

Std.  Dev. 

3.10 

6.34 

10.88 

Range 

8 

18 

40 

The  Paper  media  shows  a  real  weakness  on  this  measure  of  effectiveness. 
Subjects  using  Paper  annotated  only  56.8%  of  the  documentation  requirements  that 
were  identified  in  the  perfect  assessment.  This  suggests  that,  for  assessments 
performed  using  paper  TOs,  on  average  almost  one-half  of  all  documentation 
requirements  will  not  even  be  recorded  in  the  initial  assessment.  As  a  result,  additional 
repairs  and  documentation  requirements  will  be  identified  during  the  repair  process. 
This  is  not  due  to  any  assessor  shortcomings;  it  is  because  of  the  complexity  and 
tediousness  of  using  the  paper  TOs.  Assessors  using  COM  and  IPDF  performed  much 
higher  scores  on  this  measure.  On  average,  assessors  using  the  COM  or  IPDF  media 
types,  along  with  the  ABDAR  Demonstration  System,  will  miss  less  than  10%  of 
documentation  requirements.  Clearly,  the  electronic  technical  data,  in  conjunction  with 
the  ABDAR  Demonstration  System,  is  a  tremendous  aid  to  skilled  assessors  performing 
ABDR  assessment  tasks. 

Accuracy 

The  Accuracy  of  an  assessment  is  the  measure  of  how  accurate  the  assessor  was 
at  annotating  documentation  requirements  and  the  correctness  with  which  the 
annotations  were  assessed.  This  is  the  most  critical  measure  of  effectiveness  for  an 
assessment. 

An  assessment  can  be  done  quickly  but  poorly  (incomplete  and  inaccurate).  An 
assessment  can  be  thorough,  with  most  documentation  requirements  being  annotated, 
but  not  annotated  accurately.  Either  of  these  conditions  can  lead  to  an  inadequate 
assessment  and  additional  work  for  technicians  making  repairs  to  the  aircraft.  An 
accurate  assessment  provides  the  greatest  opportunity  for  an  aircraft  to  be  repaired  to 
provide  the  needed  capabilities. 

As  with  the  completeness  measure,  assessors  using  CDM  and  IPDF  far 
outperformed  assessors  using  Paper.  Figure  5  clearly  shows  the  performance 
advantages  provided  by  subjects  using  CDM  and  IPDF.  Assessors  using  Paper 
achieved  an  Accuracy  level  of  40%  while  assessors  using  CDM  and  IPDF  performed  at 
91%  and  83%,  respectively.  Such  a  large  margin  of  difference  between  subjects  using 
the  Paper  media  and  subjects  using  the  other  media  types  demonstrates  the 
advantages  of  the  ABDAR  Demonstration  System  in  an  overwhelming  manner.  See 
Table  7  for  means,  standard  deviations,  and  ranges  for  all  conditions. 
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Figure  5  -  Box  Plot  for  Accuracy 

(NOTE:  The  graph  shows  the  median,  inter-quartiie  range,  and  adjacent  values  for 
each  of  the  media  types  for  the  dependent  variable  of  Time.) 


Table  7  -  Description  Statistics  for  Percent  Accurate 


CDM 

I  PDF 

Paper 

Mean 

90.90 

83.30 

39.80 

Std.  Dev. 

4.43 

7.83 

12.15 

Range 

13 

23 

40 
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Resolution  of  Hypotheses 

The  following  paragraphs  provide  resolution  for  each  of  the  hypotheses  outlined  in 
the  field  test  Objectives  on  page  6.  For  convenience,  the  objectives  have  been 
restated,  in  italicized  text,  with  the  resolution  following: 

a.  The  F-15  Assessor  using  CDM  data  with  ABDAR  will  perform  significantly  better 
(fewer  errors,  less  time,  more  complete  assessment)  than  the  F-15  Assessor 
using  Paper.  This  hypothesis  was  unequivocally  supported,  F-15  Assessors 

using  CDM  performed  the  assessment  in  less  time  (x  =  338.2  vs.  x  =  479.8), 
more  completely  (x  =  96.4  vs.  x  =  60.2),  and  more  accurately  (x  =  92.0  vs.  x  = 
43.6)  than  F-15  Assessors  using  Paper. 

b.  The  Other  Assessor  using  CDM  data  with  ABDAR  will  perform  significantly  better 
(fewer  errors,  less  time,  more  complete  assessment)  than  the  F-15  Assessor 
using  paper.  This  hypothesis  was  confirmed  with  the  same  data  as  hypothesis 
number  1  because  no  significant  differences  between  Assessor  types  existed 
Other  Assessors  using  CDM  performed  the  assessment  in  less  time  (x  =  337.0 
vs.  X  =  479.8),  with  more  thoroughness  (x  =  94.4  vs.  x  =  60.2),  and  more 
accurately  (x  =  89.8  vs.  x  =  43.6)  than  F-15  Assessors  using  Paper 

c.  There  will  be  a  significant  performance  difference  (fewer  errors,  less  time,  more 
complete  assessment)  among  technicians  when  using  CDM  data  with  ABDAR. 
There  were  no  significant  differences  between  F-15  Assessors  and  Other 
Assessors  on  any  of  the  three  dependent  variables. 

d.  There  will  be  a  significant  performance  difference  (fewer  errors,  less  time,  more 
complete  assessment)  among  technicians  when  using  IPDF  with  ABDAR.  There 
were  no  significant  differences  between  F-15  Assessors  and  Other  Assessors  on 
any  of  the  three  dependent  variables. 

e.  The  F-15  Assessor  will  perform  significantly  better  (fewer  errors,  less  time,  more 
complete  assessment)  using  CDM  data  with  ABDAR  than  when  using  IPDF  data 
with  ABDAR.  The  F-15  Assessor  using  CDM  data  with  ABDAR  Demonstration 
System  performed  the  assessment  in  significantly  less  time  than  the  F-15 
Assessor  using  IPDF  (x  =  338.2  vs.  x  =  594.0).  There  were  no  significant 
differences  for  the  completeness  and  accuracy  variables. 

f  There  will  be  a  significant  interaction  between  technician  type  and  media  on  both 
errors  and  time.  CDM  data  with  ABDAR  will  tend  to  minimize  the  differences 
between  technician  types  while  the  Paper  media  with  ABDAR  will  show  large 
differences  between  technician  types.  The  interaction  will  be  ordinal  in  nature, 
meaning  the  relative  ranking  of  the  technician  types  will  remain  the  same  at  all 
three  levels  of  media.  There  were  no  significant  interactions  for  any  of  the 
dependent  variables. 
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Exit  Questionnaire  Comments  Summary 

Of  the  36  subjects  who  participated  in  the  electronic  portion  of  the  ABDAR  field  test, 
two  did  not  fill  out  the  exit  questionnaire.  The  remaining  34  subjects,  provided  answers 
to  all  four  exit  questionnaire  questions.  The  comments  by  the  subjects  provided  on  the 
exit  questionnaire  were  generally  quite  positive.  Some  system  weaknesses  were  noted 
and  several  suggestions  made  for  improving  the  system.  Several  subjects  expressed  a 
desire  to  see  the  system  implemented  for  operational  use.  The  following  paragraphs 
provide  the  questions  and  some  typical  responses  to  those  questions.  APPENDIX  P 
provides  a  complete  list  of  the  actual  written  comments  given  by  the  subjects. 


Question  1.  What  aspects  of  the  ABDAR  System  were  most  helpful  to  you  in 
performing  your  Job?  Typical  responses; 

“I  like  how  it  takes  care  of  filling  out  the  AFTO  97's.  Really  like  the  way  it 
takes  you  to  the  specific  TO  area  you  need  to  be  in,  links  are  very  useful.” 

“The  speed  in  which  it  performs  the  task.  The  way  the  screen  won't  switch 
over  unless  you  process  the  work  correctly.” 

“K-factors  automatically  figured  for  you.  No  research  required  with  this 
system.  Extremely  quick  ordering.” 

“TO's  being  in  the  computer,  reduces  a  lot  of  time  looking  through  books.” 

Question  2.  What  aspects  of  the  ABDAR  System  did  you  dislike?  Typical  Responses." 

“Not  having  detailed  parts  (information),  i.e.,  fittings  had  no  sizes  that  were 
recognizable  at  a  glance.” 

“Really  none,  just  takes  time  to  get  used  to  the  system.” 

“System  slowed  to  a  crawl  or  even  locked  up  after  a  few  hours  of  use.  I 
realize  that  the  system  is  still  under  design.” 

“When  a  qty  of  60  for  fasteners  is  needed,  then  when  you  allocate  you 
should  not  need  to  allocate  60  times  for  60  fasteners.  A  block  needs  to  be 
added  for  any  deviations  and  work  stoppage.” 

Question  3.  And  are  there  any  changes  you  would  like  to  see  made  to  the  ABDAR 
System?  Typical  responses: 

“Sensor  touch  screens  in  which  you  can  use  a  pointer.” 

“Maybe,  install  a  track  ball  type  mouse.” 

“Better  user  friendly  controls  with  voice  and  video  links  (if  possible).” 
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Question  4.  Are  there  any  other  concerns  or  comments  that  you  would  like  to  be 
known?  Typical  responses: 

“Make  one  system  and  improve  on  that  system.  TO's  should  be  all 
standardized.  Not  fighters,  bomber,  airlift,  i.e.,  if  27  is  for  flight  controls  then  it 
should  be  for  each  weapon  system.” 

“I  personally  need  more  training  to  give  a  better  viewpoint  on  what  we  really 
need.  Overall  it  is  a  vast  improvement  over  having  to  physically  search  for  info 
and  manually  document  forms.” 

“I  think  it  is  an  excellent  program.  I  think  it  should  be  helpful  to  the  assessors 
who  understand  the  setup  and  material.” 

“Make  it  so!  In  other  words  make  it  happen  as  soon  as  possible.  “ 


Table  8  provides  a  summary  of  the  responses  by  question  number,  whether  the 
response  was  positive  or  negative,  and  the  category  of  the  answers.  Also,  one  answer 
can  relate  to  more  than  one  category. 
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Table  8  -  Exit  Questionnaire  Summary 


Question  1 


Question  2 


Question  3 


Question  4 


Positive  Negative  Positive  Negative  Positive  Negative  Positive  Negative 


F  Infrastructure 


Communication 

Data 

Decision  Making 


Documentation 


Field  Test  Team 


General 
Hardware 
High  Tech 


K-factors 


Known  Problems 
Part  Ordering 
Planning 


System  Design 


TO  Research 
Usability 
View  of  Data 
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DISCUSSION 


The  ABDAR  Demonstration  System  was  tested  in  a  simulated  ABDR  environment 
and  demonstrated  improvements  in  speed,  accuracy,  and  completeness  over  the 
current  ABDR  process.  Unequivocally,  assessors  using  either  CDM  or  IPDF  media 
types  far  outperform  assessors  using  Paper.  The  only  advantage  assessors  using 
Paper  have  is  a  slight  improvement  in  time  over  assessors  using  IPDF.  However,  due 
to  the  lack  of  completeness  and  accuracy  of  the  assessments  by  subjects  using  Paper, 
the  time  advantage  becomes  irrelevant.  In  other  words,  because  the  assessments 
were  not  as  complete  and  accurate,  it  made  little  difference  how  quickly  they  were 
performed.  It  is  likely  that  any  time  advantages  gained  through  paper  assessments  are 
lost  when  the  repair  process  is  initiated  and  the  repair  technicians  must  compensate  for 
incomplete  information  provided  during  the  assessment. 

Curiously,  there  was  no  statistical  difference  between  the  performances  of  F-15 
assessors  and  other  assessors  on  any  of  the  measures  of  effectiveness.  Although,  for 
both  completeness  and  accuracy,  the  differences  observed  approached  significance. 
The  ANOVA’s  on  completeness  and  accuracy  returned  probability  values  of  .066  and 
.059  for  the  Assessor  variable  (F-1 5  Assessor  vs.  Other  Assessor).  In  both  cases,  F-1 5 
Assessors  were  more  complete  and  more  accurate.  It  is  possible  that  collecting  more 
data  would  have  revealed  a  significant  performance  advantage  for  the  F-15  Assessors, 
as  would  be  expected  for  the  assessment  of  an  F-15.  The  ANOVA’s  did  not  indicate 
any  interactions  between  assessor  type  and  media  type.  The  advantages  and 
disadvantages  of  the  media  types  were  consistent  for  both  categories  of  assessors. 

Initially  the  ABDAR  field  test  plan  provided  for  comparing  performance  data  from  7- 
level  F-15  mechanics  (crew  chiefs  and  sheet  metal  specialists)  with  the  performance  of 
the  F-1 5  and  other  aircraft  assessors.  However,  since  there  were  insufficient  F-1 5 
mechanics  available  for  the  field  test  (forcing  substitution  of  4-levels),  it  was  necessary 
to  drop  the  F-15  mechanic  data  from  the  analysis.  Since  the  F-15  data  was  collected 
and  analyzed  (Appendix  A),  an  informal  comparison  is  still  possible.  Examination  of  the 
data  provided  in  the  appendix  indicates  a  very  similar  performance  pattern  for  all  three 
variables.  Times  with  CDM  and  paper  were  significantly  faster  than  with  IPDF,  with  no 
significant  differences  between  performance  times  for  CDM  and  paper.  Completion 
and  accuracy  scores  for  CDM  and  IPDF  were  significantly  higher  than  scores  for  paper. 
There  were  no  significant  differences  in  the  completeness  and  accuracy  scores  for 
CDM  and  IPDF.  An  informal  comparison  with  the  data  for  the  F-15  and  other  aircraft 
suggests  that  there  was  little  practical  difference  in  their  performance  and  the 
performance  of  the  F-1 5  mechanics. 

The  implication  is  that  in  the  event  that  a  battle  damage  assessment  is  required  and  a 
qualified  F-15  or  other  aircraft  assessor  is  not  available,  an  F-15  mechanic  can 
effectively  perform  the  assessment  using  the  ABDAR  system.  This  could  be  an 
important  advantage  in  a  combat  situation  in  which  a  damaged  aircraft  is  forced  to  land 
at  a  base  with  no  qualified  assessors  available. 
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Analysis  of  the  field  test  results  raised  the  question  of  whether  the  advantages 
observed  for  the  IPDF  data  were  due  to  the  tools/aids  provided  by  the  ABDAR  system 
or  simply  due  to  the  fact  that  they  were  presented  electronically.  Specifically,  were  the 
beneficial  results  found  for  the  IPDF  data  due  to  the  enhancements  (e.g.,  additional 
links,)  made  to  the  data  and  the  aids  (e.g.,  wizards)  provided  by  the  ABDAR  system,  or 
were  they  due  to  the  presenting  the  information  electronically?  To  provide  some  data 
relevant  to  this  issue,  a  data  collection  team  returned  to  Robins  AFB  to  collect  data 
from  a  new  sample  of  F-15  Assessors.  An  additional  data  collection  period  was 
established  to  provide  a  tentative  answer  to  this  question.  Six  F-15  Assessors 
performed  the  assessment  task  under  the  same  test  and  evaluation  conditions  and 
rules.  For  this  test,  subjects  performed  a  conventional  assessment  with  the  use  of  Joint 
Computer-aided  Acquisition  and  Logistics  Support  (JCALS)  standard  IPDF  TOs  instead 
of  paper  TOs.  The  JCALS  IPDF  data  was  presented  electronically  with  no  additional 
enhancements  or  aids.  This  setup  allowed  collection  of  data  that  would  be  comparable 
to  the  ABDAR  enhanced  IPDF  data  and  the  paper  data  collected  during  the  field  test. 

The  results  of  this  supplemental  data  collection  effort  are  provided  in  Appendix  B. 
The  time,  completeness,  and  accuracy  data  were  consistent  with  data  collected  in  the 
main  part  of  the  study.  No  significant  differences  in  mean  times  were  found  for  any  of 
the  three  media.  The  mean  times  were  ABDAR  IPDF  -  594  minutes,  JCALS  IPDF  - 
486  minutes,  and  Paper  -  526.  For  the  completeness  measure,  subjects  using  IPDF 
scored  significantly  higher  than  both  subjects  using  JCALS  IPDF  and  subjects  using 
paper.  Subjects  using  JCALS  IPDF  scored  significantly  higher  than  subjects  using 
Paper.  The  means  were  ABDAR  IPDF  -  93.8%,  JCALS  IPDF  -  74.4%,  Paper  -  60%. 
The  data  for  accuracy  followed  a  similar  pattern.  The  mean  scores  were  ABDAR  IPDF 
-  88%,  JCALS  IPDF  -  58,  and  Paper  -  44%.  ABDAR  IPDF  subjects  performed 
significantly  better  on  the  accuracy  measure  than  subjects  using  either  JCALS  IPDF  or 
Paper.  There  were  no  significant  differences  for  the  JCALS  IPDF  and  Paper 
conditions. 

As  would  be  expected,  the  JCALS  IPDF  data  demonstrates  an  improvement  in 
performance  times  over  paper  resulting  from  easier  access  to  data  and  elimination  of 
the  need  to  physically  search  for  and  acquire  the  data.  Improvements  are  also 
observed  for  the  Completeness  variable.  The  reasons  for  this  improvement  are  not 
obvious,  but  may  be  due  to  a  greater  reliance  on  the  use  of  technical  data,  since  it  was 
more  readily  available. 

Rapid  assessment  of  a  battle  damaged  aircraft  is  important,  however,  it  is  more 
important  that  the  assessments  be  complete  and  accurate.  Comparison  of  the  field  test 
data  for  the  different  types  of  media  leads  to  some  interesting  observations.  Figure  6 
presents  a  plot  of  the  Time  data  vs  Accuracy  Data  for  the  four  media  types.  The  figure 
illustrates  the  relationship  between  three  factors,  time,  completeness,  and  cost  (where 
CDM  has  the  highest  cost,  followed  by  ABDAR  IPDF,  JCALS  IPDF,  and  Paper).  The 
best  performances  in  terms  of  accuracy  are  provided  by  the  most  expensive  types  of 
data,  CDM  and  ABDAR  IPDF.  The  less  expensive  paper  technical  and  JCALS  IPDF 
data  were  much  less  effective  for  supporting  the  ABDR  assessment  task 
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Figure  6  -  Plot  of  Time  vs.  Accuracy  for  four  Media  Types 

The  CDM  based  data  gives  the  best  overall  performance,  but  is  generally  more 
expensive.  For  new  weapon  systems,  which  will  already  have  their  technical  data  in  the 
CDM  format,  the  cost  of  adding  ABDAR  capabilities  will  not  be  great.  However,  to 
create  CDM  data  for  weapon  systems  (such  as  the  F-15)  would  be  cost  prohibitive, 
since  it  requires  completely  reformatting  and  restructuring  the  technical  data.  For  these 
weapon  systems,  the  development  of  ABDAR  IPDF  would  be  the  most  cost  effective 
solution.  The  addition  of  performance  aids  and  enhancement  of  the  existing  JCALS 
IPDF  data  would  be  relatively  inexpensive  and  would  provide  major  improvements  in 
performance. 


RECOMMENDATIONS 

The  following  section  contains  the  recommendations  of  the  ABDAR  Integrated 
Process  Team  (IPT).  Each  recommendation  is  clearly  identified  and  is  preceded  by 
supporting  arguments. 

The  performance  advantages  experienced  by  assessors  using  the  ABDAR 
Demonstration  System  with  the  CDM  and  IPDF  media  types  demonstrate  the 
advantages  that  would  be  realized  if  software  of  this  nature  were  implemented. 
Assessors  using  the  ABDAR  Demonstration  System  with  CDM  data  outperformed 
assessors  using  Paper  on  every  performance  measure  by  statistically  significant 
margins.  There  is  no  doubt  that  using  CDM  data  with  the  ABDAR  Demonstration 
System  is  a  radical  improvement  in  ABDR  methodology  compared  to  using  Paper. 
Assessors  using  the  IPDF  data  with  the  ABDAR  Demonstration  System  performed 
significantly  better  than  those  using  Paper  on  performance  measures  of  accuracy  and 
completeness.  Although  the  assessments  using  paper  were  faster,  the  benefits  of 
faster  assessments  do  not  compensate  for  the  lesser  quality  of  the  assessments. 
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Recommendation  One:  The  ABDAR  Demonstration  System  should  be 
implemented  for  support  of  USAF  ABDR  operations. 

The  single  performance  flaw  regarding  the  ABDAR  Demonstration  System  is  the 
performance  time  of  assessors  using  IPDF.  Several  options  exist  to  improve 
assessment  times  for  individuals  using  IPDF.  One  constraint  imposed  on  the  ABDAR 
program  was  to  limit  the  modifications  made  to  the  IPDF  files.  Allowing  IPDF  files  to  be 
enhanced  should  improve  the  speed  of  assessors  using  the  IPDF  files.  Also,  the  IPDF 
user  interface  is  not  easily  learned.  This  limitation  could  be  overcome  with  addition  of 
additional  features  to  make  it  easier  to  use.  The  observation  that  in  most  cases  the 
performances  on  the  second  damage  site  were  quicker,  suggests  that  more  practice 
and  training  would  reduce  the  times  required  to  perform  assessments  using  the  IPDF 
data. 


Recommendation  Two:  Efforts  should  be  made  to  improve  the  speed  with 
which  assessors  use  IPDF  media  with  the  ABDAR  Demonstration  System. 
Improving  the  IPDF  data  type,  by  linking,  and  the  improving  the  IPDF 
user-interface  should  reduce  the  time  expended  performing  ABDR  with 
IPDF. 

The  ABDAR  Demonstration  System  using  CDM  data  is  the  most  effective  tool  for 
performing  the  ABDR  assessment  task.  This  combination  provides  the  most  accurate, 
most  complete,  and  fastest  assessment  times.  CDM  media  is  the  data  type  that 
provides  the  most  robust  capability. 

Recommendation  Three:  An  ABDAR  system  using  CDM  data  should  be 
provided  for  all  future  weapon  systems.  An  ABDAR  system  using  IPDF 
data  should  be  provided  for  all  legacy  aircraft  systems. 

Implementation  of  the  above  recommendations  will  significantly  enhance  the  Air 
Force  ABDR  program.  The  course  has  been  laid  out  to  develop  a  process  (ABDAR 
Demonstration  System)  and  data  types  (CDM  and  IPDF)  that  optimize  the  assessment 
of  battle-damaged  aircraft.  The  primary  goals  of  the  ABDAR  Demonstration  System 
were  to  improve  the  speed,  accuracy,  and  completeness  of  ABDR  activities.  Following 
the  recommendations  of  this  test  report  will  allow  the  goals  of  the  ABDAR 
Demonstration  System  to  become  the  reality  of  the  ABDR  Program  Office,  AFMC,  and 
other  functional  managers. 


33 


REFERENCES 


Thomas,  D.L.  (1995)  Integrated  Maintenance  Information  System:  User  Field  Demonstration 
and  Test:  Executive  Summary.  (AL/HR-TR-1 995-0033,  AD-A293250).  Wright- 
Patterson  AFB,  OH:  Logistics  Research  Division,  Human  Resources  Directorate, 
Armstrong  Laboratory. 

Ward,  G.D.,  Kruzick,  T.M.  &  Weimer,  R,E.  (1995).  Integrated  Maintenance  Information 
System  (IMIS)  Final  Program  Report,  Volume  3:  Results,  Conclusions,  and 
Recommendations  (AL/HR-TR-1995-0035,  AD-A302291).  Wright-Patterson  AFB, 
OH:  Logistics  Research  Division,  Human  Resources  Directorate,  Armstrong 
Laboratory. 

Ward,  G.D.,  Kruzick,  T.M.  &  Weimer,  R.E.  (1995).  Integrated  Maintenance  Information 
System  (IMIS)  Final  Program  Report,  Volume  I:  Executive  Summary  (AL/HR-TR- 
1995-0040,  AD-A303609).  Wright  Patterson  AFB,  OH:  Logistics  Research  Division, 
Human  Resources  Directorate,  Armstrong  Laboratory. 

Ward,  G.D.,  Kruzick,  T.M.  &  Weimer,  R.E.  (1995).  Integrated  Maintenance  Information 

System  (IMIS)  Final  Program  Report,  Volume  2:  Program  Methodology  (AL/HR-TR- 
1995-0041,  AD-A303608).  Wright-Patterson  AFB,  OH:  Logistics  Research  Division, 
Human  Resources  Directorate,  Armstrong  Laboratory. 


34 


APPENDIX  A  -  F-16  MECHANIC  DATA 


The  F-15  Mechanic  data  is  considered  supplementary  to  the  data  presented  in  the  field  test 
report.  Not  all  of  the  F-15  Mechanics  meet  the  minimum  subject  requirement  of  being  7- 
level  qualified.  The  difference  in  experience  between  5  and  7-levels  was  deemed  to  be 
significant  and,  therefore,  the  F-15  Mechanic  data  was  be  analyzed  separately  so  as  not  to 
contaminate  the  field  test  data. 

Analysis  of  the  F-15  Mechanic  data  paralleled  the  analysis  of  the  other  data.  Three 
dependent  variables  of  Time,  Completeness,  and  Accuracy  were  measured.  The  data  from 
each  variable  were  submitted  to  a  One-Way  ANOVA,  the  results  of  which  are  presented 
below. 

The  results  of  each  of  the  ANOVA’s,  for  the  F-15  Mechanic  data,  are  analogous  to  the 
results  from  the  field  test.  Each  variable  showed  a  significant  effect  of  Media.  The  F-15 

Mechanics  Time  variable  data  (See  Table  A-1)  were  fastest  when  using  COM  (x  =  288.0), 

followed  by  those  using  Paper  (x  =  389.8)  and  then  those  using  IPDF  {x  =  580.3).  As  with 
the  Assessor  data,  Bonferroni  Multiple  Comparison  tests  showed  significant  differences 
between  subjects  using  COM  and  subjects  using  IPDF;  and  between  subjects  using  Paper 
and  subjects  using  IPDF.  There  was  no  statistical  difference  between  subjects  using  Paper 
and  subjects  using  CDM. 


Table  A-1  -  Analysis  of  Variance  Table  -  Time 


Source 

Sum  of 

Mean 

Prob 

Term 

DF 

Squares 

Square 

F-Ratio 

Level 

A:  Media 

2 

264238.1 

132119.1 

16.65 

0.000155* 

S 

15 

119026.2 

7935.078 

Total  (Adjusted) 

17 

383264.3 

Total 

18 

‘Term  significant  at  alpha  =  0.05 

The  F-15  Mechanics  Completeness  variable  data  (See  Table  A-2)  was  consistent  with  the 
Assessor  data.  Subjects  using  CDM  and  IPDF  performed  significantly  more  complete 

assessments  {x  =  95.0  and  x  =  92.8,  respectively)  than  subjects  using  Paper  (x  =  55.2). 


Table  A-2  -  Analysis  of  Variance  Table  -  Completeness 


Source 

Sum  of 

Mean 

Prob 

Term 

DF 

Squares 

Square 

F-Ratio 

Level 

A;  Media 

2 

6020.333 

3010.167 

103.64 

0.000000* 

S 

15 

435.6667 

29.04445 

Total  (Adjusted) 

17 

6456 

Total 

18 

‘Term  significant  at  alpha  =  0.05 
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The  F-15  Mechanics  Accuracy  variable  (See  Table  A-3)  was  also  consistent  with  the 
Assessor  data.  Subjects  using  CDM  and  I  PDF  performed  significantly  more  accurate 

assessments  (x  =  92.6  and  x  =  86.5,  respectively)  than  subjects  using  Paper  (x  =  33.7). 


Table  A-3  -  Analysis  of  Variance  Table  -  Accuracy 


Source 

Sum  of 

Mean 

Prob 

Term 

DF 

Squares 

Square 

F-Ratio 

Level 

A;  Media 

2 

12620.78 

6310.389 

187.01 

0.000000* 

S 

15 

506.1667 

33.74445 

Total  (Adjusted) 

17 

13126.94 

Total 

18 

‘Term  significant  at  alpha  =  0.05 
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APPENDIX  B  -  SUPPLEMENTAL  DATA 


The  original  program  plan  provided  only  for  comparing  three  types  of  data,  CDM  based, 
IPDF  based  and  conventional  paper  TOs.  However,  analysis  of  the  field  test  results  suggested 
an  additional  question,  which  required  addressing.  Were  the  beneficial  results  found  for  the 
IPDF  data  due  to  the  enhancements  (e.g.,  additional  links,)  made  to  the  data  and  the  aids  (e.g., 
wizards)  provided  by  the  ABDAR  system,  or  were  they  due  to  the  presenting  the  information 
electronically?  An  additional  data  collection  period  was  established  to  provide  a  tentative 
answer  to  this  question.  Six  F-15  Assessors  performed  the  assessment  task  under  the  same 
test  and  evaluation  conditions  and  rules.  For  this  test,  subjects  performed  a  conventional 
assessment  with  the  use  of  JCALS  standard  IPDF  TOs  instead  of  paper  TOs.  The  JCALS 
IPDF  data  was  presented  electronically  with  no  additional  enhancements  or  aids.  This  setup 
allowed  collection  of  data  that  tentatively  would  be  comparable  to  the  ABDAR  enhanced  IPDF 
data  and  the  paper  data  collected  during  the  field  test.  To  this  end,  the  data  were  submitted  to 
three  one-way  ANOVAs  to  determine  any  differences  between  the  Media  conditions  of  IPDF 
(IPDF  with  the  ABDAR  Demonstration  System),  Paper,  and  JCALS  (IPDF  without  the  ABDAR 
Demonstration  System).  As  with  the  previous  tests,  the  dependent  variables  were  Time, 
Completeness,  and  Accuracy.  Mean  scores  for  the  JCALS  IPDF  data  were  compared  with  the 
IPDF  and  paper  collected  in  the  main  field  test  for  the  F-15  assessors.  Overall  results  were 
consistent  with  the  findings  of  the  main  test. 


Time 

The  ANOVA  for  Time  revealed  no  significant  differences  between  subjects  in  the  different 
media  conditions  (See  Table  B- 1). 


Table  B- 1  -  Analysis  of  Variance  Table  -  Time 


Source 

Sum  of 

Mean 

Prob 

Term 

DF 

Squares 

Square 

F-Ratio 

Level 

A:  Media 

2 

41513.73 

20756.87 

1.02 

0.388881 

S 

12 

243511.6 

20292.63 

Total  (Adjusted) 

14 

285025.3 

Total 

15 

*Term  significant  at  alpha  =  0.05 

These  results  are  consistent  with  the  field  test.  The  significant  speed  difference  between 
IPDF  and  Paper,  observed  in  the  field  test  results  and  not  here,  is  attributable  to  the  lack  of  the 
Other  Assessor  data.  The  trend  in  the  data  is  the  same;  Paper  is  faster  than  IPDF.  The 
addition  of  the  JCALS  condition  provides  no  new  insight  into  the  data. 

Completeness 

The  results  of  the  ANOVA  for  completeness  are  shown  in  Table  B-2.  The  ANOVA  on  the 
Completeness  data  showed  a  significant  effect,  F  (2,  14)  =  31.45,  p  =  .0000.  Subjects  using 

IPDF  recorded  the  most  complete  assessments  (x  =  93.8%),  followed  by  subjects  in  the 

JCALS  condition  {x  =  74.4)  and  subjects  using  Paper  {x  =  60.2).  Bonferroni  Multiple 
Comparison  Tests  showed  significant  differences  between  each  of  the  media  types. 
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Table  B-2-  Analysis  of  Variance  Table  -  Completeness 


Source 

Sum  of 

Mean 

Prob 

Term 

DF 

Squares 

Square 

F-Ratio 

Level 

A:  Media 

2 

2844.933 

1422.467 

31.45 

0.000017* 

S 

12 

542.8 

45.23333 

Total  (Adjusted) 

14 

3387.733 

Total 

15 

*Term  significant  at  alpha  =  0.05 

As  would  be  expected,  subjects  in  the  JCALS  condition  performed  better  than  subjects  in 
the  Paper  condition  and  worse  than  subjects  in  the  IPDF  condition.  The  convenience  of  the 
electronic  TOs  improved  completeness  when  compared  to  Paper.  However,  the  combination  of 
electronic  TO’s  and  the  ABDAR  Demonstration  System  prove  to  be  the  best  method  for 
obtaining  a  complete  assessment. 


Accuracy 

The  ANOVA  on  the  Accuracy  data  (See  Table  B-3)  showed  a  significant  effect,  F  (2,  14)  = 
38.58.  p  =  .0000.  Subjects  using  the  IPDF  media  were  the  most  accurate  (x  =  87.6%) 

followed  by  subjects  using  the  JCALS  standard  electronic  TO’s  (x  =  58.2)  with  subjects  in  the 
Paper  media  condition  performing  assessments  that  were  only  43.6%  accurate.  Bonferroni 
Multiple  Comparison  Tests  reveal  significant  differences  between  the  means  of  the  IPDF 
subjects  and  both  of  the  other  media  conditions,  JCALS  and  Paper.  The  subjects  in  the  JCALS 
condition  did  not  perform  significantly  better  than  subjects  in  the  Paper  condition. 


Table  B-3  -  Analysis  of  Variance  Table  -  Accuracy 


Source 

Sum  of 

Mean 

Prob 

Term 

DF 

Squares 

F-Ratio 

Level 

A:  Media 

2 

38.58 

0.000006* 

S 

12 

781.2 

65.1 

Total  (Adjusted) 

14 

Total 

15 

*Term  significant  at  alpha  =  0.05 

Again,  the  data  analysis  follows  the  trends  present  on  the  field  test  data.  IPDF  data 
combined  with  the  ABDAR  Demonstration  system  provided  the  most  accurate  assessment. 


Summary 

The  collection  and  analysis  of  this  data  was  important  to  answer  one  question:  were  the 
performance  advantages  of  IPDF  and  CDM,  as  documented  in  the  Field  Test,  due  to  the 
electronic  TOs  or  due  to  their  integration  into  the  ABDAR  Demonstration  System? 

The  data  gathered  are  entirely  consistent  with  the  results  of  the  field  test.  They  also  seem 
to  indicate  that  a  significant  portion  of  the  performance  advantage  is  attributable  to  the  ABDAR 
Demonstration  System.  In  other  words,  IPDF  electronic  TOs  alone  do  not  provide  the  same 
advantage  that  IPDF  electronic  TOs  with  the  ABDAR  Demonstration  System  do. 
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APPENDIX  C  -  FIELD  TEST  SCHEDULE 


Table  C- 1  -  Paper  Schedule 


Paper 

Nunober 

Media 

Technician  Tjpe 

1 

M 

T 

W 

T 

F 

Weekl 

FM3 

Paper 

F-15Mech 

A 

A 

Weekl 

FM4 

F-15Mech 

A 

A 

Weekl 

Wedcl 

FA3 

OA2 

Paper 

Paper 

F-15  Assessor 
OtherAssessor 

B 

B 

B 

B 

Weekl 

OAl 

Paper 

Other  Assess  or 

C 

C 

RSUH 

FAl 

F-15  Assessor 

C 

C 

Week  2 

OA3 

Paper 

OtherAssessor 

A 

A 

Week  2 

F-15  Assessor 

A 

A 

Week  2 

bsibi 

F-15  Assessor 

B 

B 

Wedc2 

OtherAssessor 

B 

B 

Wedc2 

FMl 

F-15Mech 

C 

c 

Wedc2 

FM6 

Paper 

F-15Mech 

C 

c 

OA5 

BiTmi 

OtherAssessor 

A 

A 

Week  3 

FA4 

F-15  Assessor 

A 

A 

Wedc3 

FM2 

Paper 

F-15Mech 

B 

B 

Wedc3 

■SSSHI 

F-15Mech 

B 

B 

Week  3 

F-15  Assessor 

C 

c 

Week  3 

OA6 

Paper 

OtherAssessor 

C 

c 
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Table  C-  2  -  Electronic  One  Schedule 


Wedcl 


Week  1 


Weekl 


Week  1 


Wedcl 


Wedcl 


Nuniier 

Media 

Technician  Twe 

0A4 

IPDF 

Other  Assessor 

FA3 

CDM 

F-15  Assessor 

FM2 

COM 

F-15Medi 

FM5 

CDM 

F-ISMedi 

FAS 

IPDF 

F-15  Assessor 

OAl 

IPDF 

Other  Assess  or 

Wedc2 

0A5 

PDF 

OtherAssessor 

Week  2 

FAl5 

CDM 

F-15  Assessor 

Week  2 

FA2. 

PDF 

F-15  Assessor 

Week  2 

0A3 

CDM 

OtherAssessor 

Wedc2 

0A2 

CDM 

OtherAssessor 

Wedc2 

FA4 

PDF 

F-15  Assessor 

Week  3 

FM3 

PDF 

F-15Mech 

Wedc3 

FM6 

CDM 

F-15Mech 

Week  3 

0A6 

PDF 

OtherAssessor 

Week  3 

FAl 

PDF 

F-15  Assessor 

Week  3 

FMl 

CDM 

F-15Mech 

Week  3 

FM4 

CDM 

F-15Mech 
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Table  C-  3  -  Electronic  Two  Schedule 


Etectrordc  Two 

Number 


Weekl 

Weekl 

Weekl 

Weekl 

Weekl 

Weekl 


QAl 

FA2 

FAl 

0A2 

FM3 

FM5 


Media 

CDM 

PDF 

CDM 

PDF 

CDM 

PDF 


Technician  Twe 

Other  Assessor 
F-15  Assessor 
F-15  Assessor 
Other  Assess  or 
F-15Mecfa 
F-15Mech 


M 

_A 

A 


W 


A 


B 


T 


F 


C 

C 


Week  2 
Wedc2 
Wedc2 
Week  2 
Week  2 
Week  2 


0Ai5 

FM2 

FMl 

0A3 

FA4 

FA6 


PDF 

CDM 

CDM 

CDM 

PDF 

PDF 


Other  Assessor 
F-15Mech 
F-15Mech 
Other  Assessor 
F-15  Assessor 
F-15  Assessor 


Week  3 
Wedf3 
Week  3 
Week  3 
Wedi3 
Wedc3 


FA5 

FM4 

FA3 

FM6 

OA4 

0A5 


CDM 

CDM 

CDM 

PDF 

PDF 

PDF 


F-15  Assessor 
F-15Medi 
F-15  Assessor 
F-15Mech 
Other  Assessor 
OtherAssessor 


C 

C 
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APPENDIX  D  -  DEMOGRAPHIC  INFORMATION  FORM 


Demographics  Questionnaire 
ABDAR  Field  Test 

1.  Name: 

2.  Rank; 

3.  Stationed  at: 

4.  Age: 

5.  Sex: 

6.  Years  in  Air  Force: 

7.  Years  of  ABDR  experience: 

8.  AFSC’s: 


9.  Are  you  Color  Blind? 

Yes 

No 

10.  Do  you  have  experience  with  CAMS? 

Yes 

No 

1 1 .  Do  you  have  experience  with  CFRS  (FR/FI)? 

Yes 

No 

12.  Do  you  have  experience  with  other,  similar,  automated  systems?  Yes 

1 3.  How  would  you  rate  your  computer  skills? 

Beginner  (Little  or  no  experience) 

Average  (word  processing,  web  browser,  spreadsheet,  etc.) 

Expert  (extensive  experience,  programming,  power  user,  etc.) 

14.  Squadron  Commander’s  Name  and  mailing  address: 

15.  Your  Office  Symbol: 


No 
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APPENDIX  E  -  EXIT  QUESTIONNAIRE 
Exit  Questionnaire 
ABDAR  Field  Test 

1 .  What  aspects  of  the  ABDAR  System  were  most  helpful  to  you  in  performing  your  job? 

2.  What  aspects  of  the  ABDAR  System  did  you  dislike? 

3.  Are  there  any  changes  you  would  like  to  see  made  to  the  ABDAR  System? 

4.  Are  there  any  other  concerns  or  comments  that  you  would  like  to  be  known? 
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APPENDIX  F  -  DATA  COLLECTION  SHEETS 


Subject  Number: 

Minutes  to  T enths  of  Minutes  Conversion  | 

Session  Number: 

1  or  2  D 

21  thru  26  0.4 

46  thru  51  OB 

Session  Type: 

3  thru  8  D.1 

27  thru  33  0.5 

52  thru  57^0.9 

Monitor: 

9  thru  14  l0-2 

34  thru  39  0.6 

58  thru  BO  ^  Next  Whole 

1  Start  Time: 

IS  thru  20  0.3 

40  thru  45  0.7 

Hour 

Stop  Time: 

Total  Time: 

lUXD  Inspection  CA/V  11^ 

Inspecting  Aircraft 

START 

STOP 

HOURS 

NOTE# 

. 

Researching  Repairs  Data/Documenting 

Repair  Instructions 

. . - . . 1 

. 

START 

STOP 

HOURS 

NOTE# 

I 

•  . 

IPB  RESEARCH  TIME 

START 

STOP 

HOURS 

NOTE# 

. 

. 

Researching  Damage 
Data/Documenting  Damage 
Description 

START 

STOP 

HOURS 

NOTE# 

Repair  Planning 

START 

STOP 

HOURS 

NOTE# 

WIRING  RESEARCH  TIME 

START 

STOP 

HOURS 

NOTE# 

Note  # 

Time 

Note 
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Electronic  Assessment  Data  Collection  Form 


Subject  Number:  _ 

Session  Number:  _ 

Session  Type:  PDF  CDM 

Monitor:  i  JJ  LK 


UXO  Inspection  C/W 


UXO  Inspection  OW 


Start  Time: 
Stop  Time: 
Total  Time: 
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APPENDIX  G- SUBJECT  RULES 

The  following  rules  will  be  read  to  the  subjects,  before  the  experiment,  and  provided 
upon  request  at  any  time  during  the  experiment.  Specific  sets  of  rules  are  provided  for 
each  media  type  being  tested  as  well  as  a  general  set  of  rules  applicable  to  all  media  types. 

a.  General 

1.  Individuals  are  not  being  tested  nor  are  they  in  competition  with  any  other 
individual(s).  Different  media  types  (paper,  IPDF,  and  CDM)  are  the  focus  of  the 
study. 

2.  SAFETY  FIRST.  All  personnel  will  exercise  extreme  care  while  working  around 
the  aircraft,  due  to  sharp  edges  and  low  areas.  Caution  and  safety  will  be 
emphasized  when  climbing  on  ladders  or  working  on  top  of  the  aircraft. 

3.  There  will  be  NO  simulated  threats,  nor  will  there  be  a  requirement  to  wear  any 
protective  chemical  warfare  (CW)  gear  during  the  experiment. 

4.  Disregard  any  areas  covered  with  speed  (metal)  tape. 

5.  DO  NOT  clean  up  any  damages.  Assume  that  any  clean  up  will  be 

accomplished  as  part  of  the  repair,  not  as  part  of  the  assessment. 

6.  Assume  that  technicians  are  ABDR  trained.  This  is  for  the  sake  of 

documentation  of  the  repair  to  be  accomplished.  NOTE:  The  technicians  have 
minimum  experience  in  repairing  this  type  of  damage.  Therefore,  supply  a  more 
detailed  instruction  set.  Remember:  You  will  not  be  the  repair  technician! 

7.  DO  NOT  write  on  the  aircraft.  Use  paper  and  tape  it  to  the  aircraft  at  the 

locations  you  need  to  document.  (Due  to  varied  locations,  test  monitors  cannot 
accomplish  this  before  the  test.) 

8.  DO  NOT  write  in  the  Technical  Orders  (TOs).  If  any  marks  are  found  in  the  TO, 
point  them  out  to  the  monitor  so  they  can  be  erased. 

9.  DO  NOT  discuss  the  tasks  performed,  equipment/methods  used,  any  difficulties 
experienced,  the  specifics  of  the  study  or  anything  seen  or  heard  during  this 
experiment  with  co-workers,  other  test  subjects,  etc.  This  restriction  applies  until 
all  field  test  data  collection  has  been  completed. 

10.  During  lunch  and  other  breaks,  try  not  to  continue  to  “work”  the  session. 
Because  we  are  gathering  times  as  part  of  the  Data  Collection,  we  need  to  have 
an  accurate  measurement  of  the  time  involved  in  assessing  the  damages.  This 
includes  the  time  spent  thinking  about  the  problem. 
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b.  Paper 


1.  To  order  tools  and/or  materials  or  parts  not  available  in  the  Tools  and  Materials 
kit,  research  the  needed  information  and  fill  out  an  AF  Form  2005  or  AF  Form 
2413  and  give  it  to  the  data  collector.  It  will  be  issued  or  back  ordered  per  the 
requirements  of  the  Field  Test. 

2.  Document  information  pertaining  to  damages,  when  appropriate,  in  Blocks  16 
through  29  of  the  AFTO  Form  97. 

3.  Use  AF  Form  2406  to  indicate  all  required  Operational  Checks  in  the  planned 
sequence  for  repairs. 

4.  Use  AF  Form  2406  to  indicate  which  repairs  you  plan  to  accomplish  concurrently. 

5.  Specify  a  planned  start  and  stop  time  for  all  repairs. 

6.  Use  AF  Form  2406  to  specify  the  planned  use  of  resources, 
c.  Electronic  -  CDM  and  IPDF 

1 .  Include  all  Operational  Checks  in  repair  planning. 

2.  Order  any  needed  parts  or  materials  not  available  in  the  Tools  and  Materials  kit 
via  the  Portable  Maintenance  Aid  (PMA).  The  results  will  be  sent  via  electronic 
means. 

3.  The  assessment  portion  of  the  documentation  must  be  reviewed  by  the  test 
monitor  to  validate  that  all  damages  were  found.  This  must  be  accomplished 
before  the  session  ends. 

4.  Plan  the  use  of  resources  along  with  the  task  needed  to  accomplish  the  repair. 

5.  These  will  be  printed  out  or  saved  as  an  electronic  file.  They  will  become  part  of 
the  Data  Collection  data.  They  will  be  used  for  evaluation  and  analysis. 
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APPENDIX  H- MONITOR  RULES 

Be  ready  to  point  out  any  damage(s)  missed  when  the  subject  attempts  to  hand  in  the 
completed  assessment.  If  more  than  one  damage  is  missed,  point  out  all  missed  damages 
at  the  same  time.  Allow  them  to  complete  the  missed  damage(s).  Record  which 
damage(s)  were  missed  with  individual  time  measurements  for  those  missed. 

Record  any  help  given  and  classify  the  help.  Indicate  whether  an  explanation  was  given 
or  just  the  location  of  the  information. 

Keep  technical  interaction  with  the  subject  to  a  minimum.  Document  when  interaction 
occurs  and  what  it  entailed.  (No  social  interaction,  small  talk,  etc.  either) 

Enter  the  required  data  on  the  Data  Collection  Sheet  and  compute  total  time  for  session. 
Note  any  reason  for  work  stoppage,  including  the  relevant  times.  Things  such  as  breaks, 
lunch,  etc.  should  have  a  start  and  stop  time  noted.  (Also,  record  times  at  key  points  and 
stop/start  times  for  breaks.) 
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APPENDIX  I  -  SCENARIO  1  -  PAPER  LEFT  SIDE 


•  Scenario  1  -  Paper  Left  Side 

-  You  have  been  assigned  as  the  assessor  for  aircraft  76-0012  L  (L  is  the  left  side 
only). 

-  The  aircraft  has  been  battle  damaged  and  Debrief  has  been  accomplished  by 
another  individual  using  the  F-15  CFRS  debrief  system. 

-  The  AFTO  Forms  (97  and  781A)  have  been  annotated  with  this  debrief 
information  and  correctly  describe  the  failures  experienced. 

-  The  aircraft  is  needed  ASAP  as  lead  for  a  four  ship  Air  to  Air  night  mission,  with 
no  degraded  performance. 

-  The  aircraft  must  be  tanker  capable. 

-  You  have  a  limited  number  of  TOs  available  at  this  work  site.  Any  additional 
TOs  required  should  be  requested  from  the  Production  Superintendent. 

-  You  have  available  a  partially  stocked  war  wagon,  all  needed  tools,  all  needed 
powered  and  non-powered  Aircraft  Ground  Equipment  (AGE),  and  some 
personnel  (see  roster  below). 

-  You  have  access  to  base  supply  for  parts  ordering  and  access  to  a  Pro  Super 
for  information  and  resources. 


Personnel 

Specialty 

Special  Qualifications 

Subject  name 

Subject  specialty 

Yes 

SSgt  Miller 

Elect.  Assessor 

No 

SSgt  Smith 

SM  Assessor 

No 

SrA  Crum 

APG  ABDR  Technician 

Yes 

Eng.  &  Hyd.  Cut 

SrA  Cole 

Fuel  ABDR  Technician 

Yes 

Capt  Clark 

ABDR  Engineer 

Yes 
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APPENDIX  J  ~  SCENARIO  2  -  PAPER  RIGHT  SIDE 


•  Scenario  2  -  Paper  Right  Side 

-  You  have  been  assigned  as  the  assessor  for  aircraft  76-0012  R  (R  is  the  right 
side  only). 

-  The  aircraft  has  been  battle  damaged  and  Debrief  has  been  accomplished  by 
another  individual  using  the  F-15  CFRS  debrief  system. 

-  The  AFTO  Forms  (97  and  781A)  have  been  annotated  with  this  debrief 
information  and  correctly  describe  the  failures  experienced. 

-  The  aircraft  is  needed  ASAP  as  lead  for  a  four  ship  Air  to  Air  night  mission,  with 
no  degraded  performance. 

-  The  aircraft  must  be  tanker  capable. 

-  You  have  a  limited  number  of  TOs  available  at  this  work  site.  Any  additional 
TOs  required  should  be  requested  from  the  Production  Superintendent. 

-  You  have  available  a  partially  stocked  war  wagon,  all  needed  tools,  all  needed 
powered  and  non-powered  AGE,  and  some  personnel  (see  roster  below). 


-  You  have  access  to  base  supply  for  parts  ordering  and  to  a  Pro  Super  for 
information  and  resources. 


Personnel 

Specialty 

Available 

Special  Qualifications 

Subject  name 

Subject  specialty 

Yes 

SSgt  Miller 

Elect.  Assessor 

No 

SSgt  Smith 

SM  Assessor 

No 

SrA  Crum 

APG  ABDR  Technician 

Yes 

Eng.  Run  Qual. 

SrA  Cole 

Fuel  ABDR  Technician 

Yes 

Instruments  Cut  Trained 

Capt  Clark 

ABDR  Engineer 

Yes 

50 


APPENDIX  K  -  SCENARIO  3  -  CDM/lPDF  LEFT  SIDE 


•  Scenario  3  -  CDM/lPDF  Left  Side 

-  You  have  been  assigned  as  the  assessor  for  aircraft  76-0012  L  (L  is  the  left  side 
only). 

-  The  aircraft  has  been  battle  damaged  and  Debrief  has  been  accomplished  by 
another  individual  using  the  F-15  CFRS  debrief  system. 

-  The  paper  AFTO  Forms  781 A  and  the  electronic  equivalent  AFTO  Form  97  have 
been  annotated  with  this  debrief  information  and  correctly  describe  the  failures 
experienced. 

-  The  aircraft  is  needed  ASAP  as  lead  for  a  four  ship  Air  to  Air  night  mission,  with 
no  degraded  performance. 

-  The  aircraft  must  be  tanker  capable. 

-  You  have  available  a  partially  stocked  war  wagon,  all  needed  tools,  all  needed 
powered  and  non-powered  Aircraft  Ground  Equipment  (AGE),  and  some  ABDAR 
personnel. 

-  You  have  a  connection  to  base  supply  for  parts  ordering  and  access  to  a  Pro 
Super  for  information  and  resources. 


-  All  communications  needed  with  Pro  Super,  Team  Chief,  Engineer,  Technicians, 
Supply,  Support,  etc.  must  be  accomplished  electronically. 


Personnel 

Specialty 

Available 

Special  Qualifications 

Subject  name 

Subject  specialty 

Yes 

SSgt  Miller 

Elect.  Assessor 

No 

SSgt  Smith 

SM  Assessor 

No 

SrA  Crum 

APG  ABDR  Technician 

Yes 

Eng.  &  Hyd.  Cut 

SrA  Cole 

Fuel  ABDR  Technician 

Yes 

Capt  Clark 

ABDR  Engineer 

Yes 
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APPENDIX  L  -  SCENARIO  4  -  CDM/lPDF  RIGHT  SIDE 


•  Scenario  4  -  CDM/lPDF  Right  Side 

-  You  have  been  assigned  as  the  assessor  for  aircraft  76-0012  R  (R  is  the  right 
side  only). 

-  The  aircraft  has  been  battle  damaged  and  Debrief  has  been  accomplished  by 
another  individual  using  the  F-15  CFRS  debrief  system. 

-  The  paper  AFTO  Forms  781 A  and  the  electronic  equivalent  AFTO  Form  97  have 
been  annotated  with  this  debrief  information  and  correctly  describe  the  failures 
experienced 

-  The  aircraft  is  needed  ASAP  as  lead  for  a  four  ship  Air  to  Air  night  mission,  with 
no  degraded  performance. 

-  The  aircraft  must  be  tanker  capable. 

-  You  have  available  a  partially  stocked  war  wagon,  all  needed  tools,  all  needed 
powered  and  non-powered  Aircraft  Ground  Equipment  (AGE),  and  some  ABDAR 
personnel. 

-  You  have  a  connection  to  base  supply  for  parts  ordering  and  access  to  a  Pro 
Super  for  information  and  resources. 


-  All  communications  needed  with  Pro  Super,  Team  Chief,  Engineer,  Technicians, 
Supply,  Support,  etc.  must  be  accomplished  electronically. 


Personnel 

Specialty 

Available 

Special  Qualifications 

Subject  name 

Subject  specialty 

Yes 

SSgt  Miller 

No 

SSgt  Smith 

SM  Assessor 

No 

SrA  Crum 

APG  ABDR  Technician 

Yes 

Eng.  Run  Qual. 

SrA  Cole 

Fuel  ABDR  Technician 

Yes 

Instruments  Cut  Trained 

Capt  Clark 

ABDR  Engineer 

Yes 
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APPENDIX  M  -  EVALUATIONS 


1.  Damage  Collection  Evaluation 

a.  The  Structure  Table.  The  structure  table  contains  the  name  (type)  of  the  structure, 
the  type  of  material  it  is  composed  of,  and  the  category  of  the  structure.  Using  the  extent  of 
the  damage  (size  and  number  of  holes,  dents,  cracks,  etc.)  and  comparing  the  damage  to 
the  limits  shown  in  the  table  for  each  component,  the  Class  of  damage  can  be  determined. 
To  make  this  comparison,  a  calculation  of  a  “k  factor”  may  be  required.  From  this 
information,  the  assessor  must  document  the  damage.  The  following  is  a  suggestion  on 
how  to  evaluate  the  documentation  to  determine  if  it  meets  the  minimum,  as  described  by 
TO  1-1H-39. 

(1) .  Name:  Was  the  name  from  the  table  used?  if  not,  was  the  name  used  sufficient 
to  identify  what  component  was  damaged? 

(2) .  Location:  Was  Waterline  (WL),  Butt-line  (BL),  and  Fuselage  Station  (FS)  used  to 
determine  the  location  of  the  damage?  If  not,  was  the  location  described  using 
measurements  from  easily  identified  references?  Was  it  sufficient  in  describing  the  location 
of  the  damage? 

(3) .  Category:  Was  the  category  of  the  damage  recorded?  (To  allow  another 
assessor  to  continue,  he  needs  to  know  the  category  of  each  assessed  structure.  This 
allows  him  to  identify  the  highest  category  of  damaged  structure  and  annotate  the  AFTO 
Form  97  without  having  to  look  each  assessed  structure  up  again.) 

(4) .  Extent  of  damage:  Was  the  size  of  the  damage,  number  of  damages  included  in 
this  damage  (if  more  than  one),  and  k  factor  (if  applicable)  recorded?  Was  type  (hole,  dent, 
nick,  etc.)  of  damage  identified? 

(5) .  Class:  Was  the  Class  of  the  damage  recorded?  (The  extent  of  damage  when 
compared  to  the  table  will  give  the  Class.  The  Class  of  the  damage  is  needed  for  the  same 
reason  that  the  damage  Category  is  needed.  The  Class  identifies  the  possible  (required) 
repairs.) 

b.  The  System  Table.  The  System  Table  shows  a  name,  acceptable  damage  limits, 
possible  repairs,  method  of  determining  the  function  impact,  and  restriction/impact  if 
functional  impact  is  not  fixed.  Another  table,  in  the  1F-15A-39,  shows  serviceability  criteria 
(impact)  for  both  types  of  missions  (Air  to  Air  and  Ferry). 

(1).  Name:  Was  the  name  from  the  table  used?  If  not,  was  the  name  used  sufficient 
in  identifying  what  component  was  damaged? 
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(2) .  Extent  of  damage:  Was  damage  described  to  the  detail  needed  to  allow  another 
assessor  to  recognize  it?  Was  functionality  in  question?  If  so,  was  functionality  determined 
during  the  assessment  or  was  an  evaluation  part  of  the  repair? 

(3) .  Location:  Was  location  of  damage  on  the  system  described  in  sufficient  detail 
that  the  damage  could  be  found  by  other  ABDR  personnel? 

(4) .  Serviceability  criteria:  was  criteria  needed  for  missions  recorded? 

c.  Documentation  for  damaged  wiring  is  somewhat  more  difficult  than  damaged 
structures  and  systems.  Based  on  observation  and  feedback,  the  following  is  the  minimum 
set  of  information  needed  for  another  assessor  to  continue  an  assessment: 

(1) .  Name:  Was  the  bundle  number  or  plug  number  used  to  identify  the  damaged 
component? 

(2) .  Location:  Was  the  location  of  damaged  wiring  recorded  in  sufficient  detail  that  the 
damage  could  be  found? 

(3) .  Extent  of  Damage:  Was  the  description  of  the  damage  sufficient?  Was  the  type 
of  damage  (nicked,  cut,  etc.)  indicated?  Was  the  number  of  damaged  wires  identified? 

(4) .  Serviceability  criteria:  Were  potentially  affected  systems  noted?  Were  each 
system’s  criteria  for  missions  noted?  Were  systems  identified  that  had  to  be  fixed  and/or 
deactivated? 

2.  Repair  Instructions  Evaluation 

a.  Each  repair  must  have  a  complete  description  of  the  repair  to  be  accomplished. 
These  descriptions  shall  include  such  things  as  the  TO  figure  references,  hardware 
(material)  required,  diagrams  of  structural  patches  showing  fastener  requirements,  etc.  In 
all  cases,  the  assessor  shall  include  as  a  minimum  the  repair’s  TO  reference  (figure, 
paragraph,  etc.),  number  and  type  of  fasteners  required  (if  applicable),  and  any  peculiar 
requirements  not  contained  in  the  TO  reference.  If  an  engineer  repair  is  used,  the 
engineer’s  name  and  grade  must  be  printed  after  the  repair  instructions.  TO  1-1H-39 
contains  the  foliowing  instructions: 

(1) .  Was  the  repair  annotated  as  to  which  damage  it  fixed? 

(2) .  Was  the  proper  repair  decision  made  based  on  mission  and  serviceability  criteria? 
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(3) .  Were  resources  (parts,  material,  personnel,  etc.)  considered  in  making  the  repair 
decision? 

(4) .  Did  the  repair  contain  the  correct  TO  reference? 

(5) .  Structure:  Did  the  repair  contain  the  number  and  type  of  fasteners?  Did  it  contain 
the  type  and  thickness  of  material  to  use?  Was  a  fastener  layout  given? 

(6) .  System:  Was  a  method  given  to  determine  functional  impact  (if  in  question)? 
Was  the  appropriate  action  given  if  the  functional  impact  was  known  or  was  the  appropriate 
action  given  after  a  functional  impact  was  proven? 

(7) .  Wiring:  Were  all  affected  systems  identified  to  be  fixed  and/or  deactivated? 

3.  Repair  Planning  Evaluation 

a.  Sequencing:  Were  all  repairs  planned?  Were  all  physical  and  functional 
considerations  correctly  considered?  Were  all  resource  conflicts  correctly  considered? 

b.  Timing:  Did  the  planned  repair  times  equal  the  estimated  repair  times? 

4.  Other  Evaluations 

a.  Was  UXO  inspection  signed  off  before  starting  inspections? 

b.  Were  the  highest  Class  and  Category  noted?  Were  they  correct? 

c.  Were  estimated  times  recorded  for  damages?  Were  they  within  a  valid  range? 

5.  Overall  Evaluation  Matrix.  The  evaluation  matrix  for  Damage  Site  One  (APPENDIX  N) 
and  Damage  Site  Two  (APPENDIX  O)  show  the  evaluation  elements  common  or  unique  to 
each  type  of  damaged  component  (Structure,  System,  and  Wiring).  We  have  used  this  in 
developing  our  overall  scoring  scheme.  The  matrix  shows  each  element  is  worth  one  point 
for  each  component  within  a  damage  type.  This  means  the  total  for  the  damage  site  in  the 
Avionics  Bay  is  worth  approximately  400  points  while  the  damage  site  in  the  wing  is  worth 
approximately  200  points.  This  difference  reflects  the  difference  in  the  number  of  actual 
damages  within  each  site.  Damage  site  one  contains  14  damages  and  damage  site  two 
contains  eight. 
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APPENDIX  N  -  DAMAGE  SITE  ONE  EVALUATION  MATRIX 


Damage  Collection  Evaluation 

Inspecting  A/C  Annotated?  Accurate?  ; 

.  ; . 

Toiais 

UXO  Inspections  1;  1^ 

^Damage  Found  14 

. : .  ; 

. 16; 

Other  Documentation  Requirements 

Annotated?  Accurate? 


Highest  class? 

Most  critical  Cat? 

1:  1 

1  1 

. ; . : 

Engineer  Contacted? 

1  ^ 

Estimated  times? 

5  5 

. : . i 

16: 

Structure 

System 

Wiring 

Structure 

Annotated?  Accurate?  Annotated?  Accurate?  Annotated?  Accurate? ; 

Name 

1:  1 

Location 

1  1 

. WL  -  JL- PS  - WS . 

From  Reference 

.  . i . 

. 

. ; . 

. 

Category 

Extent 

1  1 

1:  1 

. ■ . , .  :■ 

Size 

K-factor 

Class 

1^  1 

70 

System /5JI 

Name 

.  : .  ■  . 

1 

1 

Location 

1 

i: 

Extent 

1 

1; 

Seiviceability  Criteria 

A  -  A 

1 

1; 

Ferry 

1 

1. 

50: 

Wiring  (2) 

'Name 

. : . j  . 

. 

Bundle  # 

Plug# 

Location 

. ;  . ■ . 

i:  1^ 

Extent 

. : . . . . 

. 1 ; .  1 

Senrlceability  Criteria 

. a'-  a . 

i;  i; 

Ferry 

i;  ^ 

: . 

. ^ .  ■ . 

. 

. : . 

2U. 
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Appendix  N  -  Damage  Site  One  Evaluation  Matrix  (cont.) 


Repair  Instructions  Evaluation 

Structure  System  WiliPS 

;AnnDtated?  .Accurate?  Annotated?  Accurate?  ;Annotated?  :Accurate? 


Repair  ID'ed  to  damage  1:  1:  1  1;  1 

System  Parts  Ordered  (5)  1  V  . . 


TO  Reference  or 

Engineer  Name  Requirement  1 

Structu^^^^^ 

TYpe/No.  of  fasteners 

type/thickriBss  of  rnateria^  1 ; 

Fastener  lay  out  given  ;  1  j 


System  (5) 

Fun  ct  i 0  n  a  lit  y  Eva  I  u  at  e  d ? 
Correct  Repair  Action? 


Systems/Wires  Id'ed  to  Repair 

'Fix'^ . 

Deactivate 


1' . 1  '  i; . r . 

. ; . , . . . . . . 


1 

1 


24; 

1 

. 1' . 

. ; . 

. ; 

. • 

1 

. 1; 

. : . 

■  . 

20' 

1 

. 4i 


Repair  Planning  Evaluation 


Sequenced  Considered 
Physicial 
Resources 

Planned  =  Estimated  times 


Structure  System  Wiring 

Annotated?  Accurate?  Annotated?  Accurate?  ;Annotated?  iAccurate?  i 

. 1: .  1 . 1 . 1'i .  1^  1:  2B 

1  1  1  1;  1-  1i  2B 

1:  1  1  1:  i: .  1^  2B 

. ; . : . .  ^  370 


57 


APPENDIX  O  -  DAMAGE  SITE  TWO  EVALUATION  MATRIX 


Damage  Collection  Evaluation 


Inspecting  AJC 

lAnnotated?  Accurate? 

\  Totals 

UXO  Inspections 

1 

1 

Damage  Found  DS2? 

... ; .  8: . 

. : . : 

.  :  .  '  10 

Orter  Reciuirements 

iAnnotated?  Accurate? 

Highest  class? 

1 

Most  critical  Cat? 

1  1; 

^  . 

Engineer  Contacted? 

i: 

1 

Estimated  times? 

5' 

5 

16 


Structure  System  Wiring 

Structure  -Annotated?  [Accur^e?  .Annotated?  Accurate?  ^.Annotated?  iAccurate? 

Name . . 1: . 1'; .  .  . . 

Location  1 :  1 

. . Fs'-'ws'‘ . . .  *'  .''"“''7"’ . ^'7  ..’"77'' . '.7' 7'" . 7r7 . . . . . 7"7’" . 7 . ^ . 7:7""’'."’ . "7".’’ 

From  Reference .  .  . . . ; . . 

Category . .  V . 1; . 

Extent  i  1 ;  1  i 

Size . i . ; .  . ; . ; . 

. I^^ctor .  . ; . .  . . . 

Class . ! . 1: . 1- . • . i . . 


System  (4) . | . . 

■Name  . .  . . 1/  1; . 

Location  I  ■  ■  1 . 1;  ; 

:  Extent .  .  .  . ]  . . 1; 

Serviceability  Criteria 

■'"7  a'-a . "'77 . . . '777777”'  7’'|7.777’777'’.’’’7''’'.’^7' . .7.777  77.”.’ 77 . . . ■■-■■■■■p . •■■■• . 7_'77777'7 . ■7777_ 

Ferry . .  . . .  ;  ; . 1 . . j ; . ^ . 

Wiring  . . . . . . 

^Name  .  .  : .  . .  1-  . 'l 

Bundle  # . . . | . j . ' . . . . . .  . . 

Plug  # . .  . . 

Location  1 :  1 

•Extent . i . . . . . . . 1 . 1 ; . 1 

Serviceability  Criteria 

! . ’ '7a,,7 a7' . 77.777..'..7'.' . 7.'„...  • .  .  7  ’ . ...;’.7. . 77„7.’ .7777..!... . !..!"!!.7.7..7...'...7' . !....7 . 77 . ’...L . !’ . „.!.... ..l.l!.7 . ^ . ! ’...7 . 7,.l 

Ferry  . . 7 . ’ . !'.  !7!'  . 7 . 7777 . ! . i . 7  : . ■ . . 7 . 1 
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Appendix  O  -  Damage  Site  Two  Evaluation  Matrix  (cont.) 

Repair  Instructions 

Structure  System  Wiring 

^Annotated?  Accurate?  Annotated?  Accurate?  iAnnotated?  lAccurate? 

Repair  ip'ed  to  damage  1  1  1  1!  1j  1 

Parts  Ordered  1  1  ^ 


TO  Reference  or 
Engineer  Name  Requirernent 

t y  p  e/N  a .  of  fa  st  e  n  e  rs 
T y  p  e/Th  ic  kn  e  ss  of  mat  e  ri  a  I 
Fastener  layout  given 

System  (4) 

Functionality  Evaluated? 
'Correct  Repair  Action? 


Wring  (2) 

SystemsWires  Id'ed  to  Repair 

. 

Deactivate 


1  1 


1; . 1 

1:  1 

1 . 1 


1 


4D 


1  1 

1  ■ . 1 


‘12 


16 


4 


Repair  Planning  E\^  ■  . . 

Structure  System  Wiring 

iAnnotated?  ,  Accurate?  Annotated?  Accurate?  iAnnotated?  iAccurate? 

Sequenced  Considered  ;  j  ■  i 

;  Physicial  1  j  T  f  ..  V 

Resources  1; . 1  1  1; . 1J .  1;  16; 

Planned  =  Estimated  times  1  1  1  1:  II  1  16  i 

'22Bi 
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APPENDIX  P  -  EXIT  QUESTIONNAIRE  COMMENTS  -  COMPLETE 


1  Question  1.  What  aspects  of  the  ABDAR  System  were  most  helpful  to  you  in  performing  your  job? 

Context 

Comment 

Documentation 

General 

Easy  to  find  data  when  needed.  One  touch  system. 

Documentation 

Being  able  to  compile  before  you  begin  to  fill  the  97  out  completely.  It  could  be  a  useful  tool  once  the  bugs  and 

General 

desirable  hardware  become  available. 

iiaBBi 

I  like  how  it  takes  care  of  filling  out  the  AFTO  97's.  Really  like  the  way  it  takes  you  to  the  specific  TO  area  you 

need  to  be  In,  links  are  very  useful 

General 

The  entire  system  would  be  a  good  tool  to  perform  ABDR  assessment  and  repairs. 

General 

It  made  me  realize  what  problem  areas  I  have  in  assessing  actual  battle  damage. 

General 

The  speed  in  which  it  performs  the  task.  The  way  the  screen  won't  switch  over  unless  you  process  the  work 
correctly. 

General 

The  laptop  computer  with  all  necessary  assessor  tools  loaded  on  the  hard  drive,  i.e.  (TO's,  forms).  Also  being  able 

Communications 

to  communicate  with  others  from  the  terminal. 

General 

Part  Ordering 

The  auto  catalog  system  in  the  program  and  the  parts  breakdowns. 

General 

TO  Research 

Better  assessments  can  be  made  using  ABDAR,  TO's  were  great. 

General 

The  system  was  the  most  help  in  the  inspection  and  repair  selection  area.  Also,  not  having  to  wade  through  several 

TO  Research 

TO's  helped. 

K-Factors 

TO  Research 

Part  Ordering 

K-factors  automatically  figured  for  you.  No  research  req.  with  this  system.  Extremely  quick  ordering 

K-Factors 

TO  Research 
Wizards 

Having  the  TO  in  the  computer,  k-factor  calculator  and  wizard  helped  me  greatly  because  I'm  not  an  assessor. 

integrated  tech  data  (no  big  books). 

TO  Research 

TO's  being  in  the  computer,  reduces  a  lot  of  time  looking  through  books. 

TO  Research 

Electronic  versions  of  the  tech,  orders  and  the  outlined  figure  blocks  that  linked  me  to  the  tech.  data. 

TO  Research 

"Online"  tech  data. 

TO  Research 

Being  able  to  go  to  the  TO  quickly  to  get  the  Information  needed. 

TO  Research 

The  elimination  of  paper  tech  orders. 

TO  Research 

Information  about  each  repair  installed  on  the  computer. 

The  graphics  really  helped. 

Electronic  TO  figures  on  the  computer  with  part  numbers. 

TO  Research 
Documentation 

It  made  it  easy  as  far  as  CAT  damages,  and  less  paper  work. 

TO  Research 

Part  Ordering 

The  menu  for  where  the  aircraft  damage  was.  Not  having  to  look  up  part  #’s. 

TO  Research 

Convenience  of  references.  Accurate  tracking  of  jobs  and  resources.  Will  enforce  "completeness"  in  training  as 

Planning 

General 

well  as  on  the  job  assessments. 

TO  Research 
Usability 

SPEED.  Not  having  to  research  through  different  TO’s  and  wiring  diagrams  saved  loads  of  time.  EASE  OF  USE 

TO  Research 

Having  a  diagram  and  a  complete  breakdown  of  damaged  area.  Wiring  breakdown  was  very  helpful.  Automatically 

Wiring 

Decision  Making 

deciding  if  engineer  was  necessary  was  good. 

TO  Research 
Wizards 

Wizard,  TO’s  on  the  computer. 

Usability 

Set  up  like  windows,  which  makes  it  easy  to  learn. 

Usability 

Reducing  the  task  of  carrying  (physically)  TO’s  around  the  work  areas. 

Usability 

Portability  of  the  laptop. 

Usability 

System  is  portable  right  to  the  aircraft.  This  made  it  easy  to  compare  the  graphics  to  the  actual  damaged  areas. 

Wiring 

Being  able  to  have  the  wiring  diagram  at  a  click  of  a  button! 

Wizards 

ABDAR  Wizard  and  ABDAR  Manual. 

Wizards 

It  didn’t  let  you  miss  anything 
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Question  2.  What  aspects  of  the  ABDAR  system  did  you  dislike? 

No  TO's  in  the  CDM. 

Data 

Lack  of  reference  data. 

Data 

Not  having  detailed  parts,  i.e,,  fittings  had  no  sizes  that  were  recognizable  at  a  glance. 

Data 

It  is  too  vague  in  the  actual  repair  areas. 

General 

None 

Genera! 

None 

General 

I  liked  the  system.  Keep  fine-tuning  and  improving  your  product. 

General 

None 

General 

All  features  seemed  very  worthwhile. 

General 

None 

General 

Really  none,  just  takes  time  to  get  used  to  the  system. 

General 

None 

General 

None 

General 

None 

General 

None 

General 

Not  being  totally  comfortable  with  ABDAR  and  computers. 

Hardware 

Wizard 

Finger  mouse 

Known  Problems 

When  selecting  materials  etc.  It  not  being  in  alpha  order. 

Known  Problems 

There  were  a  few  kinks  with  server,  which  happens. 

Known  Problems 

TO  pages  are  hard  to  read,  when  you  enlarge  its  easy  to  lose  your  spot. 

Known  Problems 

System  slowed  to  a  crawl  or  even  locked  up  after  a  few  hours  of  use.  I  realize  that  the  system  is  stili  under 
development  though. 

Known  Problems 

I  Nothing  about  the  system.  I  would  like  to  see  the  hourglass  to  let  me  know  the  computer  is  working. 

Known  Problems 

The  speed  of  the  computer,  in  a  real  situation,  it  is  possible  to  have  a  great  deal  more  damage.  This  will 
require  a  faster  and  more  reliable  system. 

Known  Problems 
Hardware 

System  Design 

Speed,  Information  had  to  be  entered  more  than  once  and  the  touch  sensor  mouse. 

Planning 

Showing  the  repair  planning  was  not  need  for  assessor  to  his  Job  would  save  time  if  this  went  to  Job  Control  or 
Pro  Super. 

System  Design 

Having  to  allocate  twice  instead  of  once  for  resources. 

System  Design 

The  jumping  between  different  screens. 

System  Design 

Repetitiveness  -  having  to  send  a  low  pack  to  the  aircraft  10  separate  times. 

System  Design 

The  scheduling  and  repetitiveness  with  repair  selection  in  the  resource  fields. 

System  Design 

Repetitive  system. 

System  Design 

The  system  was  often  slow  at  times. 

System 

Design 

Known  Problems 

If  a  UXO  is  found  during  area  UXO  inspection.  General  UXO  and  Safe  for  Maintenance  Inspection  should  be 
opened  back  up. 

Color  of  all  scroll  bars  should  be  gray  not  blue  -  whatever  (should  stand  out  better). 

Hardware 

If  a  repair  description  is  not  complete  and  you  want  to  move  to  another  repair,  you  should  be  allowed  to  get 

back  to  the  original  repair  and  pick  back  up  where  you  left  off.  Not  start  all  over  again.  Touch  pad  is  bad! 
Should  be  mouse  or  track  ball.  Not  touch  pad!!  Wizard  is  sometimes  a  bother.  It  should  be  an  option  not  a 
must  use.  The  more  advanced  you  get  the  less  one  is  to  use  any  wizard.  Data  transmission  is  very  slow. 
Needs  to  be  faster  for  the  use  to  be  useful. 

More  training  time. 

Wizards 

The  wizard  was  a  little  confusing  at  times  but  it  helped  me  a  lot. 

Question  3.  Are  there  any  changes  you  would  like  to  see  made  to  the  ABDAR  System? 

Data 

More  selections  of  materials  and  equipment. 

General 

Looks  good! 

General 

No 
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General 


General 


Genera! 


Hardware 

Data 


High  Tech 


Implementation 


Known  Problems 
Known  Problems 

General _ 

Planning 


System  Design 


Planning 
System  Design 


Planning 
System  Design 


Planning 
System  Design 


System  Design 


System  Design 
System  Design 


System  Design 


System  Design 


System  Design 


System  Design 


System  Design 
High  Tech 


Usability 


View  of  Data 


Sensor  touch  screens  in  which  you  can  use  a  pointer. _ 

Maybe,  install  as  track  bail  type  mouse.  _ 


Maybe  if  computers  were  faster.  Some  items  in  equipment  were  actually  materials  and  vise  versa. 


Incorporate  digital  camera  so  you  can  take  a  snapshot  of  damage  and  have  available  if  needed,  this  would 
prevent  returning  to  the  aircraft  as  much.  _ _ 


Implement  [integrate?]  this  system  to  IMDS.  _ 


Have  the  program  written  in  a  more  stable  version  of  JAVA.  _ _ 


Work  out  the  bugs! 


Task  scheduling  is  not  a  needed  feature.  Allocating  resources  is  not  a  need  In  the  field. _ _ 


When  a  qty  of  60  for  fasteners  is  needed,  then  when  you  allocate  you  should  not  need  to  allocate  60  times  for 
60  fasteners.  A  block  needs  to  be  added  for  any  deviations  and  work  stoppage. 


I  would  like  to  be  able  to  transfer  equipment  and  materials  from  the  needed  to  the  allocated  block  in  repair 
planning  component.  _ _ 


I  would  like  to  see  the  scheduling  go  away.  More  of  the  management  level  tool,  than  lower  level. 
Once  the  assessor  is  complete  the  team  chief  should  schedule  people. 

None  other  than  if  it  could  all  be  put  on  one  screen. 


I  know  it  will  be  enhanced  in  the  future  and  will  become  more  user  friendly. _ 

More  user  friendly,  I.e.  hyperlinks.  _ 


Drop  and  drag  features;  better  way  to  highlight  comments  and  restrictions;  possibly  like  web  links. _ 

Give  equipment  list  a  memory.  _ _ _ 


Some  kind  of  warning  to  contact  an  engineer  if  damage  limits  are  exceeded. _ 


Areas  where  the  "double  click"  option  could  be  used  are  in  the  equipment,  personnel  and  material  selections. 
Some  areas  where  you  could  answer  more  than  one  question  on  one  screen. _ 


Better  user  friendly  controls  with  voice  and  video  links  (if  possible). 


More  training  time _ _ _ 

If  possible,  to  create  an  exploded  view  of  all  damage  area’s  (like  the  wiring  diagram)  to  zoom  in  with. 


Add  a  pictorial  feature  for  viewing  damages.  _ 


Better  graphics  _ _ 


Question  4.  Are  there  any  other  concerns  or  comments  that  you  would  like  to  be  known? 

AF  Infrastructure  Make  one  system  and  improve  on  that  system.  TO's  should  be  all  standardized.  Not  fighters,  bomber,  airlift, 

i.e.,  if  27  is  for  flight  controls  then  it  should  be  for  each  weapon  system. 

Data  1 

Having  ail  TO's  listed  in  each  system  TO  (-39)  as  well  as  peacetime  books  in  the  computer  (or  on  disk, 
internet)  will  assist  with  fixing  all  aircraft  in  our  inventory. 

Field  Test  Team 

1  would  like  to  express  my  sincere  appreciation,  for  the  kindness,  courtesy,  and  tremendous  support  the  entire 
ABDAR  team  provided! 

Field  Test  Team 

The  training/experiment  staff  was  extremely  courteous  and  helpful  and  really  makes  using  this  system  an 
easy  task.  Very  professional  image. 

General 

Overall  it  looks  like  a  good  system  in  work,  and  should  be  used,  not  only  ABDR,  but  in  regular  maintenance 
as  well.  1  think  that  a  different  program  should  be  made  for  each  different  AFAC  (shop).  In  the  future  of 
course. 

General 


Neat  system,  think  it  would  work  fine  on  any  of  our  aircraft  now. 
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General 

None 

General 

1  personally  need  more  training  to  give  a  better  viewpoint  on  what  we  really  need.  Overall  it  is  a  vast 
improvement  over  having  to  physically  search  for  info  and  manually  document  forms 

General 

No 

General 

No 

General 

No 

General 

No 

General 

Awesome  product,  it  was  a  lot  of  fun! 

General 

1  learned  a  lot  and  hope  the  Air  Force  buys  this  program. 

General 

None 

General 

None 

General 

1  think  It  is  an  excellent  program.  1  think  it  should  be  helpful  to  the  assessors  who  understand  the  setup  and 
material. 

General 

No 

General 

None 

General 

Make  it  so!  In  other  words  make  it  happen  as  soon  as  possible. 

General 

None 

General 

No 

General 

1  think  this  is  an  outstanding  program.  This  will  make  my  job  as  an  assessor  easier.  Not  to  mention  the  time 
factor  in  returning  the  aircraft  back  to  service. 

General 

Very  helpful  system  to  the  assessor. 

General 

The  new  system  is  great.  It  will  save  time  and  money  in  the  long  run. 

General 

Other  than  the  known  bugs  it’s  a  great  system. 

General 

No 

General 

Documentation 

1  enjoyed  this  test.  It  broke  down  what  one  thinks  and  puts  it  on  paper.  This  way,  things  have  a  less  chance 
of  being  overlooked. 

System 

Design  Planning 

Scheduling  the  jobs  would  be  easier  on  paper. 

Usability 

1  think  the  ABDAR  system  as  it  is  right  now  will  give  too  much  flexibility  to  the  technician  doing  the  repair  and 

It  could  cause  an  inadequate  repair  being  installed  on  the  aircraft. 

Usability 

Hardware 

Make  it  easier  for  a  non-sheet  metal  person  to  be  able  to  identify  rivets,  metals,  etc.  1  worry  about  how  long 
the  laptop  will  last  on  a  flightline  environment.  (Being  dropped,  impractical).  How  will  the  system  work  in  a 
chemical  environment  with  chemical  gear  on? 

View  of  Data 

Measurement  data:  While  most  is  very  similar,  there  are  differences  that  can  change  the  status  of  damage  if 
not  computed  correctly.  Perhaps,  have  the  ABDAR  wizard  reference  this  data  for  the  particular  aircraft  being 
assessed. 

View  of  Data 

It  was  a  bit  difficult  to  specify  an  exact  wire  or  hydraulic  line  in  a  particular  area.  1  can’t  imagine  how  much 
more  difficult  it  will  be  if  ALL  the  lines  and  ALL  the  wiring  were  to  be  added.  It  would  definitely  take  a  person 
with  experience  on  this  airframe  to  pinpoint  specific  wires  or  lines  once  they  all  get  added,  or  even  a  specialist 
on  that  system. 

View  of  Data 

If  the  wizard  followed  automatically  would  be  helpful  if  computer  could  automatically  list  type  of  material  used 
in  the  area  it  is  told,  i.e.,  type  of  metal  and  thickness  of  the  door  on  F-15  nose. 
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